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Executive Summary 



The National Network of Eisenhower Regional Consortia and Clearinghouse has become a 
dynamic local, regional, and national school improvement support system for mathematics and 
science. Funded in 1992 by the U.S. Department of Education (ED), Office of Educational 
Research and Improvement (OERI) to help improve mathematics and science education 
throughout the Nation, each of the ten Eisenhower Regional Consortia serves an ED regional 
educational laboratory region. They are assisted and supplemented by the Eisenhower National 
Clearinghouse (ENC) located at The Ohio State University. The working relationships and 
infrastructure that ENC and consortia have built with each other and hundreds of partners across 
the country have produced a mechanism for effectively delivering services and products to extend 
mathematics and science reform. 

The consortia are specifically charged to: (1) provide technical assistance for the implementation 
of teaching methods and assessment tools, for use by elementary and secondary students, teachers, 
and administrators; (2) disseminate exemplary mathematics and science educational materials; 
and (3) build networks among mathematics and science resources within their regions and 
nationally. The clearinghouse collects, describes, and disseminates K-12 mathematics and 
science materials on an extensive scale nationally. The consortia and the clearinghouse, 
individually and as a collaborative network, conduct an intensive internal evaluation to document 
and assess their activities and impacts, and, just as importantly, to provide information for 
strategic decision-making. This internal evaluation effort has been supplemented by several 
third-party evaluations sponsored by ED and reported separately. 

A. Network Evaluation 

This evaluation report focuses on the results of in-depth interviews conducted in Fall 2000 to 
determine the impacts of training and technical assistance, dissemination, and networking and 
collaboration activities. It includes summary descriptive information of Eisenhower Network 
services, products, and partnerships over the 1995-2000 five-year period. It also describes 
outcomes in terms of the OERI performance indicators, as required by funding regulations. The 
primary data sources are a Consortia and Clearinghouse Descriptive Data System (CCDDS), 
which documents all activities in terms of content and participants, and surveys and interviews, 
which gather qualitative assessments from clients. These data sources are part of the internal 
evaluation procedures. 

B. Highlights of Findings 

This five-year evaluation shows that the Eisenhower Network has provided substantial support 
for improvement in mathematics and science education to educators across the country. The 
Network focuses on three major service areas: training and technical assistance, dissemination, 
and network-building. Over the five years, services were delivered principally through 169,798 
in-person client contacts and 72,491,957 contacts using print and Internet media. 

1. Training and Technical Assistance 

The CCDDS documented large-scale client participation in training and technical assistance 
services. There were 147,271 contacts with clients through in-person consortia training and 
technical assistance activities during the 1995-2000 grant period. These services focused on local 
schools; approximately three-fourths of the clients worked in schools, and most of these were 
teachers. The Network intended to increase the intensity of these experiences for participants 
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over the years, and the proportion of activities that exceeded 12 hours in duration increased from 
39 percent in 1998 to 51 percent in 2000. 

Clients rated the quality and impact of these services highly through surveys. At least 97 percent 
reported that the technical assistance was aligned with state and national standards. Over 91 
percent reported improvements in practice as a result, and over 89 percent reported improvements 
in student performance. In addition, at least 70 percent of the participants were from high-needs 
schools, a key focus of the Eisenhower Network. 

Recent in-depth client interviews have reinforced these findings. For example: 

The consortium staff was able to demonstrate and model effective practice by working 
individually and collectively with teachers to give them a better understanding of how to 
use manipulatives in the teaching of math. That is the type of environment I think makes 
for helping at-risk students in their growth. 

Elementary Principal 

I received the Connected Mathematics Project training provided by the consortium. The 
professional development was phenomenal. One professional development program we 
went to was two and a half days, 18 hours of high energy. It was packed with neat things 
you could take back and use to engage your kids. You just wanted to start the next week 
and go ! 

Math Teacher 

Certainly students’ academic achievement is enhanced. Last year was the first year, and I 
know it’s partly because of my involvement with the consortium and my study group that 
I’m seeing these improvements in students’ performance. Our building scored the 
highest on the state science tests out of all six elementary schools in our city. My kids, 
for the first time in years, couldn’t wait to get to science class! 

Elementary Teacher 



2. Dissemination 

Widespread and effective dissemination of high-quality materials was documented. In all, 
72,491,957 contacts with clients were recorded in dissemination activities over the five years. 
This includes 2,423,754 contacts through print materials, 67,350,561 through Web sites, and 
2,717,642 through electronic listservs. The Eisenhower Network met and exceeded its goal of ten 
percent increases in these combined media contacts each year. 

Clients rated the quality of the products highly, as demonstrated by survey results. At least 68 
percent of the responding clients indicated that they used these resources and that they were of 
value to their work. 

The client interviews added further details, indicating that unique resources were provided, 
collaborative product development was supported, product dissemination was enhanced, and the 
products changed practice. For example: 
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The information from the consortium has helped me do my work more effectively. It 
helps me to be knowledgeable about what’s going on in other states and what best 
practices there are, and those types of things. It helps me do my job better. 

LEA (Local Education Agency) Science Coordinator 

The consortium is a link to national-level products on mathematics and science 
education. Serving as an ENC Access Center has allowed us access to quality materials 
for my use and in my training sessions with educators. 

TA (Technical Assistance) Provider 

If you have a question, they find you the answer. The consortium connects us to 
resources and keeps us informed. 

LEA Curriculum Director 

The resources are helpful and informative. The quality of the publications has been 
beneficial to teachers. 

LEA Supervisor 



3. Collaboration and Networking 

The consortia and clearinghouse have successfully leveraged resources through extensive 
collaborative efforts. Networking and collaboration also have helped develop the capacity of the 
collaborators. Services in this area involved 22,527 client contacts over the five years. In 
addition, at least 82 percent of all Eisenhower Network activities involved collaborative partners. 

The results of client surveys indicated that substantial value was added to the collaborators as a 
result of their participation in Network activities. At least 80 percent of survey respondents 
reported strengthened relationships among collaborators, increased coordination in providing 
services, increased access to resources, efforts leveraged for greater impact, and assistance in 
carrying out work more effectively. 

The client interview results also spoke about these impacts, for example: 

The niche filled by the consortium is their ability to convene and connect stakeholders to 
work collaboratively on systemic reform. The consortium has also been able to connect 
key organizations within the state in a way that no state-specific organization can do. 

The state department can’t bring all the key players together. They have tried. Only the 
consortium has been able to do this successfully. 

TA Provider 

As an intensive site, we got to network with other teachers from our state and from all of 
the consortium schools at one state meeting and two regional institutes. It was really 
great to get together with the state team and share ideas. . .it was wonderful just to share 
success stories, or, you know, what didn’t work. 

Elementary Teacher 

Last spring, I attended a two-day workshop on Case Studies so that I could share that 
approach to ongoing professional development with the teacher leaders I mentor. It was 
wonderful to have an opportunity to participate with other professional developers, or 
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people who work on a larger scale than just the school or district scale, from other parts 
of New England. In that way, the consortium holds a unique role, in that there isn’t a lot 
out there for support and professional development for the professional developers. You 
know, it’s an isolated bunch. And it’s those networking opportunities that make it 
possible for all of us to do our jobs better. 

SEA (State Education Agency) Staff 



C. Conclusions 

These findings were interpreted as strongly supporting the significant scale and value of 
Eisenhower Network services to clients throughout the country. In addition to relatively 
intensive resources provided in-person to a large number of teachers and others in the 
mathematics and science community, the Network has established Internet and print media 
capable of reaching all stakeholders. 

The application of stringent Performance Indicators gave another view of a capable and 
productive Eisenhower Network. As seen in the report, the Network has successfully met nine of 
the ten benchmarks for indicators that are currently measured. 

In addition, clients have indicated that Network services are unique in four ways: providing 
national and regional support, building capacity through professional development, forging 
connections across boundaries, and serving high-needs schools. The challenges evident in the 
evaluation results provide direction for future development, such as continuing to enhance the 
intensity of professional development experiences. 

A final significant strength of the Eisenhower Network is demonstrated by the existence of the 
comprehensive and detailed information available for this report. From the inception of the 
consortia and clearinghouse, the Network members have been committed to self-evaluation for 
improving their services to clients. The Network has invested substantial resources in designing 
and conducting a coordinated evaluation effort. This coordinated evaluation, in turn, reflects the 
way the Network’s 1 1 member organizations work closely together to develop, disseminate, and 
coordinate resources for improving mathematics and science education in the nation’s schools. 
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I. Introduction 



The National Network of Eisenhower Regional Consortia and Clearinghouse has become a 
dynamic regional and national school improvement support system for mathematics and science. 
Funded in 1992 by the U.S. Department of Education (ED), Office of Educational Research and 
Improvement (OERI), to help improve mathematics and science education throughout the Nation, 
each of the ten Eisenhower Regional Consortia (referred to as “consortia” throughout this report) 
serves an ED regional educational laboratory region. They are assisted and supplemented by the 
Eisenhower National Clearinghouse (ENC, also referred to as the “clearinghouse”) located at The 
Ohio State University. The working relationships and infrastructure ENC and consortia have 
built with each other and hundreds of partners across the country have produced a mechanism for 
effectively delivering services and products to extend mathematics and science reform. 

The consortia are specifically charged to: (1) provide technical assistance for the implementation 
of teaching methods and assessment tools for use by elementary and secondary students, teachers, 
and administrators; (2) disseminate exemplary mathematics and science education materials; and 
(3) build networks among mathematics and science resources within their regions and nationally. 
The clearinghouse collects, describes, and disseminates K-12 mathematics and science materials 
on an extensive scale nationally. The consortia and the clearinghouse, individually and as a 
collaborative network, conduct an intensive internal evaluation to document and assess their 
activities and impacts, and, as importantly, to provide information for strategic decision-making. 
This internal evaluation effort has been supplemented by several third-party evaluations 
sponsored by ED and reported separately. 

A. Purpose, Audience, and Use 

This evaluation report is addressed primarily to sponsors and partners of the consortia and 
clearinghouse network, as well as the staff of these organizations. It focuses in-depth on 
training/technical assistance, dissemination, and collaboration/networking activities conducted 
throughout the consortia five-year grant, and contains other selected summary analyses. It 
describes the activities and impact of the network, and also highlights areas for improvement of 
services. The Eisenhower Network published descriptive annual reports in 1997 and 1998 that 
depict activities anecdotally and include limited data about the type and number of activities and 
participants, or clients, nationally. The Network also has published previous evaluation reports 
for 1996-97 (May 1998), 1998 (May 1999), and 1999 (June 2000). These evaluation reports 
augment the annual reports by including more comprehensive, detailed descriptive information 
and participant opinion data about the usefulness, quality, and impact of Network services, 
products, and partnerships. The evaluation reports also describe outcomes in terms of the OERI 
performance indicators, as required by funding regulations. 

The major purpose of this evaluation effort is to assist the ten individual consortia, the 
clearinghouse, and the Network to reflect on their work in order to improve their practice and 
impact. The combined use of an ongoing Consortia and Clearinghouse Descriptive Data System 
(CCDDS) database, in effect since October 1 995, and more recently developed participant (client) 
survey and interviews provide rich data for understanding and assessing the work of the consortia 
and clearinghouse. Specifically, the evaluation report is intended to describe the client service 
activities across all ten consortia and the clearinghouse, document the contribution to the 
improvement of mathematics and science education by assessing client perceptions of quality and 
impact, and stimulate recommendations for improving services and the evaluation itself. Also, 
consortia partners and relevant policymakers may find the report useful in determining their 
desired role in connection with the consortia and clearinghouse to support the improvement of 
student performance in mathematics and science. 
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B. Sources of Data 

The ten Eisenhower Regional Consortia and ENC have been involved in sustained efforts as a 
national network to define and collect pertinent data that describe their services to clients and 
assess their effectiveness. To that end, in 1993 the Network established a standing evaluation 
committee that prepared a national evaluation plan including both descriptive and assessment-of- 
quality components. That plan was put into place in October 1994 by launching a pilot-test year 
of quantitative data collection using the CCDDS (Consortia and Clearinghouse Descriptive Data 
System). The data system was extensively revised based on the pilot test, and the refined system 
has been in place since October 1995. The database describes each client-service activity 
conducted by the Eisenhower Network in terms of participant affiliation and role (e.g., teacher, 
administrator, policymaker), the length of the activity, type of service provided, content focus, 
contact method, and involvement of collaborators. 

In 1997, a preliminary version of a national participant survey was developed and piloted in order 
to enact the assessment-of-quality component of the evaluation plan. The survey was designed to 
gather outcome and impact data from the recipients of products and services, with particular 
interest in effects on professional practice and student performance. Revised versions of the 
survey were administered in 1998 and 1999. 

In 2000, an in-depth telephone interview procedure was developed and implemented in order to 
collect more qualitative information on impact. In all, 163 interviews were completed across the 
Network in the Fall of 2000. The results were coded for access by theme and used as the primary 
source for this report. The principal impacts noted by respondents are discussed and illustrated 
extensively by quotation from clients. 

It should be noted that data for the 1995-2000 period for the Northwest Region was provided by 
the Columbia Education Center, which operated the Consortium in this region during that time. 
Starting in October 2000, the Northwest Regional Educational Laboratory took over operation of 
the Northwest Consortium and, consequently, responsibility for its evaluation. 

More information on the evaluation methodology is given in Appendix A. The client interview 
protocol used in 2000 is presented in Appendix B. Appendix C contains an external audit report 
on the interview process. Appendix D shows the data collected for the CCDDS. 

C. Highlights of Findings 

This five-year evaluation shows that the Eisenhower Network has provided substantial support 
for improvement in mathematics and science education to educators across the country. The 
Network focuses on three major service areas: training and technical assistance, dissemination, 
and network-building. Over the five years, services were delivered principally through 169,798 
in-person client contacts and 72,491,957 contacts using print and Internet media. 

1. Training and Technical Assistance 

The CCDDS documented large-scale client participation in training and technical assistance 
services. There were 147,271 contacts with clients through in-person consortia training and 
technical assistance activities during the 1995-2000 grant period. These services focused on local 
schools; approximately three-fourths of the clients worked in schools, and most of these were 
teachers. The Network intended to increase the intensity of these experiences for participants 
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over the years, and the proportion of activities that exceeded 12 hours in duration increased from 
39 percent in 1998 to 5 1 percent in 2000. 

Clients rated the quality and impact of these services highly through surveys. At least 97 percent 
reported that the technical assistance was aligned with state and national standards. Over 91 
percent reported improvements in practice as a result, and over 89 percent reported improvements 
in student performance. In addition, at least 70 percent of the participants were from high-needs 
schools, a key focus of the Network. 

Recent in-depth client interviews have reinforced these findings. For example: 

The consortium staff was able to demonstrate and model effective practice by working 
individually and collectively with teachers to give them a better understanding of how to 
use manipulatives in the teaching of math. That is the type of environment I think makes 
for helping at-risk students in their growth. 

Elementary Principal 

1 received the Connected Mathematics Project training provided by the consortium. The 
professional development was phenomenal. One professional development program we 
went to was two and a half days, 18 hours of high energy. It was packed with neat things 
you could take back and use to engage your kids. You just wanted to start the next week 
and go! 

Math Teacher 

Certainly students’ academic achievement is enhanced. Last year was the first year, and I 
know it’s partly because of my involvement with the consortium and my study group that 
I’m seeing these improvements in students’ performance. Our building scored the 
highest on the state science tests out of all six elementary schools in our city. My kids, 
for the first time in years, couldn’t wait to get to science class! 

Elementary Teacher 



2. Dissemination 

Widespread and effective dissemination of high-quality materials was documented. In all, 
72,491,957 contacts with clients were recorded in dissemination activities over the five years. 
This includes 2,423,754 contacts through print materials, 67,350,561 through Web sites, and 
2,7 1 7,642 through electronic listservs. The Eisenhower Network met and exceeded its goal of ten 
percent increases in these combined media contacts each year. 

Clients rated the quality of the products highly, as demonstrated by survey results. At least 68 
percent of the responding clients indicated that they used these resources and that they were of 
value to their work. 

The client interviews added further details, indicating that unique resources were provided, 
collaborative product development was supported, product dissemination was enhanced, and the 
products changed practice. For example: 

The information from the consortium has helped me do my work more effectively. It 
helps me to be knowledgeable about what’s going on in other states and what best 
practices there are, and those types of things. It helps me do my job better. 

LEA (Local Education Agency) Science Coordinator 




3 



15 



The consortium is a link to national-level products on mathematics and science 
education. Serving as an ENC Access Center has allowed us access to quality materials 
for my use and in my training sessions with educators. 

TA (Technical Assistance) Provider 

If you have a question, they find you the answer. The consortium connects us to 
resources and keeps us informed. 

LEA Curriculum Director 

The resources are helpful and informative. The quality of the publications has been 
beneficial to teachers. 

LEA Supervisor 

3. Collaboration and Networking 

The consortia and clearinghouse have successfully leveraged resources through extensive 
collaborative efforts. Networking and collaboration also have helped develop the capacity of the 
collaborators. Services in this area involved 22,527 client contacts over the five years. In 
addition, at least 82 percent of all Eisenhower Network activities involved collaborative partners. 

The results of client surveys indicated that substantial value was added to the collaborators as a 
result of their participation in Network activities. At least 80 percent of clients reported 
strengthened relationships among collaborators, increased coordination in providing services, 
increased access to resources, efforts leveraged for greater impact, and assistance in carrying out 
work more effectively. 

The client interview results also spoke about these impacts, for example: 

The niche filled by the consortium is their ability to convene and connect stakeholders to 
work collaboratively on systemic reform. The consortium has also been able to connect 
key organizations within the state in a way that no state-specific organization can do. 

The state department can’t bring all the key players together. They have tried. Only the 
consortium has been able to do this successfully. 

TA Provider 

As an intensive site, we got to network with other teachers from our state and from all of 
the consortium schools at one state meeting and two regional institutes. It was really 
great to get together with the state team and share ideas. . .it was wonderful just to share 
success stories, or, you know, what didn’t work. 

Elementary Teacher 

Last spring, I attended a two-day workshop on Case Studies so that I could share that 
approach to ongoing professional development with the teacher leaders I mentor. It was 
wonderful to have an opportunity to participate with other professional developers, or 
people who work on a larger scale than just the school or district scale, from other parts 
of New England. In that way, the consortium holds a unique role, in that there isn’t a lot 
out there for support and professional development for the professional developers. You 
know, it’s an isolated bunch. And it’s those networking opportunities that make it 
possible for all of us to do our jobs better. 

SEA (State Education Agency) Staff 
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D. Conclusions 



This evaluation effort was undertaken to enable description of the activities, outcomes, and 
impacts of the Eisenhower Network effort, as well as to highlight areas for improvement and 
change. Overall, the evaluation findings from the CCDDS, client survey, and client interview 
indicate that the consortia and ENC have taken great strides toward enabling effective reform of 
mathematics and science education. The Network has delivered a variety of client-service 
activities to intended audiences numbering in the millions, and consortia and ENC products and 
services have been rated very positively by clients. 

The evaluation results provide a picture of what has been accomplished over the five-year grant. 
In the coming months, the consortia and clearinghouse staff will focus on how to use the findings 
to inform strategic decision-making which will shape future client-service efforts. 

E. Future Evaluation Activities 

The Eisenhower Network evaluators and directors have agreed to continue to use their two 
primary data collection sources: the CCDDS and the participant surveys and interviews. A short, 
formative report based on CCDDS data will be produced for the first half of each fiscal year 
during the summer. A detailed, more summative report will be produced after the end of each 
fiscal year during the following winter. 

Participant surveys, interviews, and/or observations of activities will be conducted by the 
Network every year. Clients and activities to be sampled will be representative of all consortium 
and clearinghouse services. Case studies of sites receiving intensive services will be initiated in 
200 1 . It is anticipated that evaluation activities also will be undertaken by external evaluators 
under ED funding Network evaluators will cooperate with these external efforts and utilize the 
results as appropriate. 

The evaluation committee will continue to meet at least twice annually. The primary purpose of 
the winter meeting will be to produce a national-level report based on the prior year’s data. The 
primary purpose of the summer meeting will be to review evaluation procedures and results, and 
to design changes in the evaluation system based on report results, current priorities, and 
currently available data. Additional meetings will be scheduled for special purposes as needed. 

F. Structure of this Report 

The sections that follow begin with a brief summary of the Eisenhower Network activities and 
outcomes in Section II. Section IE highlights in-depth findings in the three major activity areas 
of the Network: training and technical assistance, dissemination, and collaboration and 
networking. Section IV focuses on the implications of the findings. The Appendices provide 
details concerning the evaluation methodology, the client interview form, the audit report on 
these interviews, the CCDDS content, the OERI Performance Indicator results for 1999-2000, 
and a matrix showing the relationship of the interview vignettes included in the report to the 
performance indicators. 
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II. Summary Description of Network Activities and Outcomes 



As indicated in the introduction above, the Eisenhower Network has three principal purposes: to 
provide technical assistance and training that will help practitioners implement improved 
practices; to disseminate exemplary mathematics and science education materials that are useful 
in the classroom; and to build networks that help support improvement. This section provides a 
very brief summary of the scale of services provided to address these purposes over the five-year 
period from October 1995 through September 2000. It also depicts the results of these services as 
measured against the performance indicator benchmarks that have been established by ED for 
reporting on the Network’s progress to Congress. 

A. Training and Technical Assistance 

Training and technical assistance are the most intensive services provided by the Network; they 
are provided primarily by the Eisenhower Regional Consortia. In general, such services are 
intended to support improved classroom practices and, as a consequence, student achievement. 
These services are designed to meet high-priority needs in the client environment, and they utilize 
high-quality materials and approaches. Training tends to engage larger groups of participants in 
building knowledge and experience, while technical assistance helps smaller groups of clients 
solve a focused problem or plan a particular event or resource material. Training and technical 
assistance services are combined in Network evaluation reporting. 

Table 1 depicts the scale of training and technical assistance services in terms of the numbers of 
client contacts by year. Overall, 147,271 client contacts took place through network services 
under this purpose. Many activities were conducted for multiple days, engaging educators in in- 
depth work. The scale of service is quite substantial, especially given the modest level of funding 
available to the consortia. 



Table 1 

Eisenhower Network Client Contacts 
In-Person Services 



^--Activity 


Training and 


Network 


Year*^'"~~^ 


Technical Assistance 


Building 



1996** 


31,630 


5,060 


1997** 


31,630 


5,060 


1998 


33,869 


4,479 


1999 


28,336 


4,753 


2000 


21.806 


3.175 


Total 


147,271 


22,527 



*Data are based on Consortia fiscal years, e.g., 1996=October 1995 through September 1996. 

**Data for these two years were combined in a two-year CCDDS set which was divided in half 
for this display. 
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Over the five years, the scale of client contacts decreased. This is a function of two factors. The 
consortia aimed to increase the duration of the activities to provide more intensive experience for 
participants, and activity length did increase substantially over the years. The consortia made a 
conscious strategic choice to increase the quality of these services, based on the research on best 
professional development practices, with a consequent sacrifice in quantity. At the same time, 
the funding for the consortia remained constant, thus inhibiting growth. Even at the lowest level 
in 2000, the scale of training and technical assistance services provided by the Network is 
notable. 

B. Dissemination 

The dissemination purpose is realized on a very large scale as shown in Table 2. The Eisenhower 
Network utilizes both print and electronic media to deliver informative resources to clients across 
the country. These resources include newsletters, curriculum materials, thematic reviews, multi- 
media professional development tools, and forums for interactive discussion. 



Table 2 

Eisenhower Network Client Contacts 
Large-Scale Dissemination 



^^Activity 

Year*^^^^ 


Consortia 


Print 

ENC 


Consortia Web 


Electronic 
Consortia Listserv 


ENC Web 


1996 


NA** 


NA 


NA 


NA 


NA 


1997 


306,557 


NA 


NA 


NA 


NA 


1998 


340,185 


170,337 


780,259 


685,000 


13,801,102 


1999 


125,212 


553,861 


2,108,797 


1,220,049 


16,394,536 


2000 


129,961 


797,641 


2,872,240 


812.593 


31.393,627 


Total 


901,915 


1,521,839 


5,761,296 


2,717,642 


61,589,265 



Across the Eisenhower Network, 2,423,754 print products were disseminated, and 70,068,203 
client contacts were made electronically through Web sites and listservs. This level of 
dissemination shows the capacity of the Network to reach the entire potential client pool in 
mathematics and science education using the media developed during 1995-2000. 

As seen in Table 2, the clearinghouse is responsible for a large majority of the Web contacts, 
although all Network members have a substantial Web presence. It should also be noted that the 
consortia and clearinghouse actively collaborate on large-scale dissemination media. 

Over the five years, electronic dissemination has seen exceptional growth, more than doubling in 
scale since data are first available across the Network in 1998. Although during the same time 
consortia print dissemination decreased, clearinghouse print increased. This yielded an overall 
increase for print media as well. 



*Data are based on Consortia fiscal years, e.g., 1996=October 1995 through September 1996. 
♦♦Dissemination data are not captured consistently across the Network in the CCDDS in 
several cells, which are marked NA. 
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C. Collaboration and Networking 

Table 1 also shows the scale of services devoted to network building. These services facilitated 
collaborative activities and resources to build capacity for mathematics and science education 
improvement. Examples include supporting advisory and professional development groups, 
sponsoring conferences, and building communications networks. While networking is important 
per se as a service to clients, it is probably even more important as a strategy to increase 
improvement efforts by engaging a wide variety of partners and resources. 

Over the five years, 22,527 client contacts took place in the context of in-person network-building 
services. As significant, over the last three years, 82 percent to 90 percent of all consortia 
activities involved collaborating partners (Table 3). The Eisenhower Network strongly 
emphasizes networking and collaboration as means to effective improvement in mathematics and 
science education. 

D. OERI Performance Indicators 

The Eisenhower Regional Consortia and the Eisenhower National Clearinghouse have standards 
for their individual productivity and performance built into their program plans. These standards 
cover the nature, scale, and quality of services intended by each organization. As a Network, the 
consortia and clearinghouse have agreed to a set of performance indicators, promulgated by OERI 
and used by the Department of Education in its performance reports to Congress since 1998. 

These indicators set 1 1 benchmarks for the consortia concerning key features of technical 
assistance, collaboration, and dissemination services. Achievement on the benchmarks is 
measured using the CCDDS, client surveys administered by Network evaluators, and third-party 
evaluations commissioned by the Department of Education. The results for 1998, 1999, and 2000 
are given in Table 3. (More detail is given specifically on the 2000 results in Appendix E.) 

Of the ten indicators currently measured, eight benchmarks have been met consistently across the 
Network in all years, and a ninth has been met in one year. Among these, the degree of positive 
findings have generally increased over the years. These results attest to both the high quality and 
scale of services provided. 

One indicator benchmark has not been met in any year (Indicator 1.2). The consortia aim to have 
60 percent of their activities be intensive, meeting at least a standard of 12 hours. The results 
have come close but not met the benchmark. Performance on another benchmark, effectively 
training trainers (Indicator 1 .6), was under in one year and above the criterion in another, and 
therefore is not consistent. The Indicator 1.4 on student test scores has not been measured yet, 
and so is marked “NA.” It will be measured in 2001. Also, six of the other indicators were 
marked “NA” for 2000 because they are measured by the client mail survey, which was not 
administered in 2000. The consistent high results in previous years suggests that these indicators 
would have been met in 2000 also. 

In summary, all these findings show an efficient and effective program of Eisenhower Network 
services. The next section provides an in-depth view based on interviews of clients. 
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Table 3 

Eisenhower Network Performance Indicator Summary 
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III. Outcomes and Impacts 



This section of the report focuses on results derived from the 163 in-depth client telephone 
interviews conducted by the Eisenhower Network evaluators. Please see Appendix A Evaluation 
Methods, for more information about how interviews were conducted. This appendix also 
includes descriptive information on the interviewees. The purpose of these interviews was to 
collect more detailed qualitative feedback from clients than was possible using the previous 
larger-scale mail surveys. The benefit of this methodology is that it provides in-depth descriptive 
information about the quality and utility of Consortia work. The limitation of using this method is 
that the information is obtained from a small sample. Therefore the information may, or may not, 
generalize to the total population. 

The interviews have been analyzed thematically, rather than quantitatively. The themes are 
illustrated with extensive quotations, or vignettes, drawn from the interview transcripts and 
organized under the three Network purposes: training and technical assistance, dissemination, and 
collaboration and networking. The vignettes are numbered in order of presentation and cross 
referenced to the OERI Performance Indicators in Appendix F. While the clients’ own words 
have been preserved as much as possible, quotations have been edited for clarity. 

A. Training and Technical Assistance 

The training and technical assistance activities of the Eisenhower Regional Consortia were 
designed to improve participants’ classroom practice and, consequently, student engagement and 
performance. These activities typically included a focus on the alignment of curriculum, 
instruction, and assessment practices with state and/or national standards. Previous Network 
evaluation reports have relied upon quantitative data to document these activities and, to some 
extent, their impacts. This section presents vignettes created from in-depth client interviews 
conducted in the Fall of 2000 to provide more qualitative data. The vignettes illustrate the unique 
role that the consortia provide with respect to training and technical assistance, the scope and 
nature of these services, and the impact on various clients. 

1. Overview of Impact 

With regard to technical assistance and training, the consortia play a valuable role in several 
ways. They offer a wide range of expertise in curriculum, instruction, and assessment in 
mathematics and science education; immediate access to the latest information and research- 
based resources in mathematics and science education; and an understanding of the systemic 
nature of educational reform and how that plays out at the local, state, regional, and national 
levels. In addition, the consortia are in touch with the needs and concerns of educators within 
their regions. Using this information, they are able to tailor the training and technical assistance 
they provide to meet individual client needs. 

A member of a state coalition for mathematics and science in the Mid-continent Region expressed 
appreciation for the valuable overall role of the consortia and the expertise that they bring to staff 
development: 

I think the consortium fills a very important niche, for all school districts who take advantage 
of it. The consortium staff has a wide variety of expertise — professional expertise in the area 
of math and science — that you can’t find any place else. Additionally, when I am looking for 
something very specific, it is not ok for them to say, “We will get back to you,” and six 
months later they haven’t. With the consortium, it is usually right on the phone, they will say, 
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“Talk to so and so in this city, or that city” or “You know there is a teacher in-state doing the 
exact things you are looking for, why don’t you give them a call?” Then you just pick up the 
phone and call these people and say, “Gee, I got your name from somebody at the 
consortium,” and they say, “Oh yeah, I know those people.” It just kind of opens the door and 
breaks down the barriers and off you go. What seemed like an insurmountable problem ten 
minutes ago is now an easily answered question. I have also gotten information, either printed 
or electronically, or the consortium has given me a web site to go check out — just all sorts of 
valuable things that I really appreciate. 

— Vignette # 1 

In providing training and technical assistance, the consortia work collaboratively and flexibly 
with schools, tailoring assistance to meet individual school needs. At the same time, consortia 
assistance is grounded in research and focused on aligning practice with state and national 
standards. An elementary school principal from the North Central Region commented on the 
ways in which the consortium staff tailored staff development by helping his staff to align state 
goals with the standardized achievement test used statewide and by assisting teachers in 
improving their practice using research-based strategies. The result was a better learning 
environment for underrepresented students and improved student performance: 

The consortium staff has been providing intensive technical assistance in the area of 
mathematics to us for the past two years. Specifically, the consortium worked with our 
staff during the half-day in-services on the alignment of state goals and objectives to the 
Iowa Test of Basic Skills and in effective, research-based instructional strategies. One of 
the areas that we noticed our teachers were weak in was the use of manipulatives and the 
instruction of math. The consortium staff was able to demonstrate and model effective 
practice by working individually and collectively with teachers to give them a better 
understanding of how to use manipulatives in the teaching of math. That is the type of 
environment I think makes for helping at-risk students in their growth. The consortium’s 
approach was tailored to the school’s needs, as opposed to a “canned” program; that’s 
what made the difference. We went up a great deal on our math test scores in the two 
years that the consortium has been working with us. 

— Vignette #2 

Focusing on the consortium’s understanding of issues related to systemic reform, a professor in 
higher education within the Southwest Region talked about what she, and other participants, 
gained from attending a regional meeting sponsored by the consortium: 

We would all get a common understanding of issues related to systemic reform. We get 
current on what is coming up, what is important to each of us, what is important for the 
students, and what is important to the government. 

— Vignette # 3 

Because the consortia provide tailored, research-based expertise that is focused on mathematics 
and science school reform, the technical assistance and training provided through the Eisenhower 
Regional Consortia have had a positive impact on teachers, students, teacher preparation 
programs, informal education institutions, districts and states, and training-of-trainers programs. 

Each is discussed below. 
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2. Impact on Teachers 



Evidence of the impact of technical assistance and training on teachers included an increase in 
teacher knowledge and skills, increased confidence in teaching mathematics and science, 
increased collegial sharing among teachers, and improvements in classroom practice. 

• Teacher knowledge and skills increased. A consultant with the state department of 
education in the Southeast Region, who is also a content specialist at a local school, first became 
involved in 1992 with the consortium as a participant in the Academy — a “trainer-of-trainers” 
model for the improvement of mathematics and science education. Since then, she has not only 
improved her own teaching but also has done a tremendous amount of consulting work related to 
consortium activities, particularly with regard to the National Science Education Standards, 
Project 2061, and Alternative Assessment. This client improved her own practice by 
implementing standards-based curricula, instruction, and assessment in her classroom. She 
contends that: 

With the training that I received over the years through the consortium, the Academy, the 
various conferences, promising practices, etc., it came home to me that my focus had not 
always been on learning, but to keep students’ attention. However, my keeping students’ 
attention wasn’t as important as focusing their attention on the objectives. I came to 
understand that, while I wanted interesting activities, I wanted interesting activities that 
had a purpose, that were focused on the standards and the objectives that I was trying to 
convey to the students. And I think, if nothing else, in my classroom during the time I was 
affiliated with the consortium, it was that involvement that made me not just a better 
science teacher, a more enjoyable science teacher for my students, but it made the 
substance of my content stronger. 

— Vignette #4 

In addition, this teacher’s efforts for improving her teaching resulted in being recognized by the 
National Board for Professional Teaching Standards. The Board rewarded her instructional 
excellence: 

Had I not had the background, the training, the exposure to what was going on outside 
my state and around the country with science and math education reform, I would not 
have known about National Board certification. And, I feel relatively certain I would 
never have achieved national certification had I not had the kind of exposure that the 
professional development and training through my affiliation with the consortium has 
afforded me. 

— Vignette #J 

Several of the interviewed teachers described a similar experience. For example, one teacher from 
the Southwest Region explained: 

Last year, I was a presidential awardee. I really credit my work with the consortium and 
the experiences that she provided for us through the Master Teachers project for helping 
me to get that award. I credit the training I got through the consortium with making me 
aware of national standards and TIMSS. Before being involved with the Master Teachers 
project I had heard of national standards but I had not worked with them a whole lot. 

— Vignette #6 
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Several teachers throughout the consortia regions have received recognition for excellent 
instruction. They attributed that accomplishment to their participation in consortia professional 
development activities that provided them increased knowledge and skills. 

• Teacher confidence increased. In addition to increasing their knowledge of instructional 
practices, several of the interviewed teachers also commented on their increased confidence in 
teaching mathematics and/or science because of their involvement in consortia activities. An 
elementary school teacher in the Northeast and Islands Region described how her school’s 
involvement with the Regional Alliance Consortium affected her confidence as a teacher: 

Math has always been one of the areas that I’ve done, but not felt really comfortable 
about doing. Three years ago, when we changed over our math program to the Everyday 
Math, it was quite frightening. The first year we taught it, we were like, “Oh my 
goodness. We can’t do this. We need some help.” That’s when we turned to the 
consortium. Having these people come in and help us through this certainly built up my 
confidence tremendously. The program was so new to us, and many of the teachers 
haven’t had geometry or algebra or those kinds of things in many, many years. And to be 
expected to teach it, when you’re not feeling comfortable with it to begin with, was a 
difficult task. Having the people come in and actually present a lot of the new strategies 
for us was very helpful. I now have a better knowledge base and a lot more self- 
confidence in teaching math. 

— Vignette #7 

• Teacher collegial sharing increased. Several of the interviewees attributed an increase in 
teachers’ collaboration and sharing to the training they had received from their regional 
consortium. A principal of a middle school involved in the Appalachia Region’s Middle School 
Mathematics Project shared his views of how the consortium’s technical assistance improved his 
mathematics teachers’ performance: 

This school has a renewed focus on planning to ensure that the curriculum is aligned with 
the standards and that the instruction and assessment are aligned with the curriculum. In 
addition to the improvements in instructional practice, I am seeing a collegiality among 
the math teachers. Before our involvement in the Middle School Math Project, it was a 5 th 
grade teacher in one room and another 5 th grade teacher in another room. The 6 th grade 
teacher didn’t talk to any of the other teachers. Now, teachers are collaborating and 
sharing. Before, teaching and learning were rarely discussed among teachers because of a 
fear factor — you might not be viewed as capable among your colleagues. The work with 
the consortium has given the math teachers here permission to talk and share about 
teaching and learning. From this discussion came our movement toward better alignment 
of curriculum, instruction, and assessment. 

— Vignette # 8 

• Classroom practice improved. The impact of the regional consortia on classroom practice 
included improving the alignment of curriculum, instruction, and assessment with standards; 
increased use of inquiry-based and hands-on activities; increased use of alternative forms of 
assessment; increased attention to multiculturalism; and increased reflection on the process of 
teaching. The following interview segments illustrate the increased use of these types of 
practices. 

A district science supervisor in the Mid-Atlantic Region noted how participation in consortium 
activities helped to better align professional development opportunities in their district with 
standards: 
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The two TIMSS conferences sponsored by the consortium allowed me to introduce TIMSS 
into the Project 2061 workshop and to connect the TIMSS results to the science benchmarks. I 
would never have done that if I had not gone to the TIMSS conferences. 

— Vignette # 9 

In the Northeast and Islands Region, an elementary school teacher also commented on how the 
consortium assisted with aligning curriculum and instruction with standards. Over a two-year 
period, her school was designated as an intensive school site, where the school team received 
more than 60 hours of professional development, coaching, and technical assistance focused on 
their mathematics curriculum and instruction, plus two, three-day summer institutes: 

During the first math course offered by the consortium, we had to design a specific unit 
that had to be aligned with the standards. That was really the first time that I’ve ever had 
to really look at our state standards and align a unit that I’ve developed, especially in the 
math area. We looked at all the different state standards and went through some of the 
activities that would tie into that standard, some of the hands-on type activities that I 
really wasn't familiar with. By doing them, practicing them, and manipulating some of 
the hands-on type materials, I was able then to take those materials back into the 
classroom and know that I was hitting the standards that I was supposed to be hitting with 
those activities. 

— Vignette #10 

Another elementary school teacher in the Northeast and Islands Region noted how her new 
understanding of the state standards allowed her to implement them in a way that met the needs 
all students, even those who struggle with reading, through the use of instructional activities that 
incorporate different modalities: 

I had always had a tendency to go by the science text. Now, I've realized that the science 
text does not follow the standards so much. You know, there’s a unit on digestion and 
there’s a unit on plants, etc., but the state standards are more geared to assimilating 
information and gathering data and collecting facts. Now in my classroom, we’re doing a 
lot more data collection, we’re doing a lot more assimilation of information, we’re doing 
a lot more writing with science, as opposed to just, “Here's the text. Read the chapter. 

What do you think?” Additionally, I’m learning — because of my experience with 
consortium — to vary the way that I teach so that all modalities are reached, instead of just 
the regular ways that I used to do. They had to leam how to collect data, so they had to 
leam how to read a tape measure instead of a text, or they had to leam how to use a scale 
or read millimeters. For the kids who are non-readers, all of a sudden, they were able to 
participate more actively because the reading part was out of the way. 

— Vignette #11 

In addition to learning about the alignment of instruction with standards and using varied 
interactive instructional practices, one higher education teacher in the Southwest Region talked 
about the value she gained from a regional meeting that focused on how to meet the multi-cultural 
needs of students within education: 

The training was helpful in providing me with an awareness of some of the multi-cultural 
issues that are out there. For me, the major minority in my area is Black. I had taught in a 
90 percent Black school. I did not realize many of the needs of those students until I 
actually was immersed in their culture during the meeting — for a very short period of 
time — through other people in the room. There was quite a diverse representation in the 
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room and we were able to become aware of needs that we had not even thought about — 
especially for me, with the Native American population and the Hispanic population. We 
don’t have many of these minorities in my area, but you move over two parishes and 
there you are. We have student-teachers who leave here and go teach in those parishes. 

The training not only made me more aware but it gave me some ideas to give my student- 
teachers — ideas on how to better serve their students. 

— Vignette #12 

Finally, in addition to learning more effective instructional and assessment practices, the training 
and technical assistance prompted several teachers to describe how they are being more reflective 
about their instructional practice. An elementary teacher in the Far West Region said: 

In Science Cases, we have collegial conversations once a month with fellow teachers and 
consortium facilitators, where we reflect on and analyze the way we teach certain 
concepts. Our conversations focus on the metacognitive side of teaching. For me, it’s 
helped in thinking about how I'm going to improve my teaching in a way that enhances 
student outcomes. It’s also how to think about and how to approach it so I CAN do it. 

The case studies make me far more aware of what has to be done, and how it has to be 
done sooner and better. I’ve realized that I need to expose my students to certain concepts 
so they have a foundation to build on in the later grades. I’ve learned that I have to 
change how I approach teaching. The case study discussions have affected my mental 
approach, and how I set up what I’m doing with the children. I like to see this sort of 
professional development being offered, because this is where the change is going to 
happen. I’m not going to change when you run me through some two-hour workshop and 
I come away with eight dittos and a game. I need to analyze what I do. I can get value out 
of the science case discussions. ..THIS stuff, I can reflect on! 

— Vignette #13 

3. Impact on Students 

The interviews with consortia clients also served to illuminate the relationship between changes 
in teacher practice and improvements in student performance. In this instance, student 
performance is broadly defined to include what students know (i.e., achievement), what they can 
do (i.e., demonstration of higher-order thinking skills and other skills emphasized in standards- 
based education), and the extent to which students are actively engaged and interested in the 
study of mathematics and science. 

• Student achievement improved. Although the evidence was often anecdotal, consortia 
clients did comment on the positive association that developed between working with their 
regional consortia and improvements in their students’ test scores. One administrator attributed 
the continuing rise in students’ mathematics and science test scores to teachers’ attendance at 
consortium-sponsored workshops in the Appalachia Region. Mathematics teachers in the North 
Central Region also felt that the increase in test scores for their underrepresented students was 
due to their collaborative work with the regional consortium around teacher professional 
development: 

I have been participating in Project REAL (Rural Education Alignment for Learning) the 
last two years. Project REAL involves six very poor, rural school districts. These school 
districts were chosen to be a part of the project because of their high poverty levels and 
because they have been classified as academic-emergency or academic-watch schools. The 
consortium is one of the partners of this project and has been providing professional 
development and networking opportunities. As a result of my participation in Project 
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REAL, I have developed a class called Math Labs that’s hands-on, engaged learning. I 
don’t sit at the desk anymore. Instead, I go outside and we apply math and do some things 
outside of the classroom. I think that is a big reason why our state test scores really jumped 
up like they did and our kids showed improvement in the areas of measurement, geometry, 
and problem solving. I give a lot of credit to the involvement I have had with the 
consortium. We were first in our county and we were one of the highest scorers in the state. 

I think that’s because of some of the neat ideas, the motivation, and the sharing with other 
math teachers and with the people at the consortium around the curriculum changes we 
made. We have a couple of fourth grade teachers on Project REAL and their math scores 
went up 20 percentage points over this past year. They really think that it has to do with 
Project REAL. 

— Vignette # 14 

A science teacher in the Mid-Atlantic Region provided professional development to other 
teachers using support from the regional consortium. In describing the process and its effects on 
student engagement and student achievement, the teacher explained: 

We take high-tech equipment to the classroom and use it to co-teach with high school science 
teachers. Our mission is the professional development aspect of it. To get the “goodies” in the 
classroom, teachers have to attend one of our workshops during the summer. We go into all 
but one high school across the state: rural, urban — even prisons. We send out evaluations after 
our classroom visits, and teachers say they are trying new things and changing their 
instruction. Students use laptops interfaced with electronic data collecting probes. The kids 
say, “Hey this is cool,” and they are up and engaged. We have done our own pre- and post- 
tests and an independent evaluator did an evaluation. His results supported the conclusion that 
the project has enhanced achievement for students. 

— Vignette til 5 

• Student engagement enhanced. Clients also mentioned improvements in students’ interest 
in learning mathematics and science. In the Northeast and Islands Region, a study that compared 
200 teachers who participated in professional development with a random selection of non- 
participants found significant differences in student engagement, curiosity, and active 
participation. An elementary teacher from the same region also commented on students’ high 
level of engagement with the standards-based science kits: 

Kids will ask to come in at recess to do some more tries with the kits we use to do some 
of the physical science. . .you know, gears, levers, pulleys that the kids use to build 
models. They’re talking about what they’re doing and going above and beyond what they 
have to do. 

— Vignette ttl6 

In the Pacific Region, a state mathematics curriculum specialist, who has been involved with the 
consortium since it began, has received reports of students taking the initiative to engage in 
mathematics activities during their free time. Additionally, the curriculum specialist has received 
reports about increased student attendance in schools where the teachers are implementing the 
standards-based practices they have learned through the regional consortium training: 

I work with teachers, students, and school administrators in facilitating and coordinating 
the mathematics curriculum across schools. I’ve been with the consortium from the 
beginning and we’ve made great strides. I’ve seen definite changes in math and science 
classrooms. More lessons are standards based, with teachers doing more inquiry-based 
activities. As a result, students spend more time learning from one another within groups. 
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Students definitely are more engaged and motivated to learn math! Because of all of the 
math activities that have been provided and the training received through the consortium, 
all of our students K-12 are enjoying their math classes. In fact, they love mathematics 
more than any of their other subject areas. I have received numerous phone calls from 
principals regarding positive changes in the math classrooms: attendance in class has gone 
up, and in one class, students have requested to play math activities during recess time! 
Because of the consortium training, students and teachers are changing their views of 
mathematics from thinking of it as boring and difficult to thinking of it as a fun subject. 

— Vignette #7 7 

Teachers are seeing students become more engaged by activities that are hands-on. As one 
elementary school teacher in the Northeast and Islands Region shared, making science come alive 
through hands-on activities ultimately has an effect on student performance: 

In past years, I’ve noticed that kids do poorly in science and social studies as a rule 
because they don’t find it interesting. Now, science is like a living organism in my 
classroom. Students are not trapped by the book. They can move around. They get to sort 
and classify and measure and weigh. And I don’t know of any kid who wouldn’t find it 
interesting if they were getting out on the playground and measuring for a playground, or 
measuring a ramp and how far the ball went. When they take part in an activity where 
they are in charge of writing down the information with a partner, they own the 
information. They take responsibility for it and they understand it a lot better than if 
they’re getting it just straight from the book. So, certainly their academic achievement is 
enhanced. Last year was the first year, and I know it’s partly because of my involvement 
with the consortium and my study group that I’m seeing these improvements in students’ 
performance. Our building scored the highest on the state science tests out of all six 
elementary schools in our city. My kids, for the first time in years, couldn’t wait to get to 
science class! 

— Vignette # 75 

Helping teachers to implement more culturally relevant curriculum in the Mid-continent Region 
was another way that the work of the consortia supported activities that engaged both students 
and parents in learning mathematics and science: 

I teach at an elementary school where the majority of students are Native American and 
considered at-risk. We were looking at where our students were lacking in meeting the 
standards and made a commitment to emphasize those areas in ways that highlighted the 
students’ culture. As part of my involvement in the Culturally Relevant Curriculum 
program, we were able to help connect what students were learning in science to their 
culture. Both the students and their parents became more engaged in studying science. 
Students were much more excited about studying the stars and the planets when we 
connected it to their culture. Then they would sit and really listen. We also offered family 
science nights to get parents involved. Working with the consortium was sort of a 
catalyst; I don’t think I would have done any of this if I hadn’t had the training or been 
encouraged by my school to go ahead and do it. And I am sure that none of these 
activities would have been done in our school even though our students are 
predominately Native American; we just did not have that much awareness of Native 
American culture in the school. Since we started working with the consortium, however, 
we have really connected the students’ culture with math and science and the students are 
benefiting as a result. 

— Vignette # 19 
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4. Impact on Teacher Preparation 



The impact of the consortium goes beyond the K-12 classroom. Teacher preparation programs are 
also rethinking how they go about their work. For example, a high school mathematics teacher in 
the Southeast Region has been involved in consortium-supported activities for almost eight years. 
She has been a participant in the Academy (a “trainer-of-trainers” model for the improvement of 
mathematics and science), the Summer Institute (which provided a model for teaching/leaming 
mathematics topics in pre-algebra, algebra, trigonometry, pre-calculus, calculus, and linear 
algebra), and several other consortium-sponsored workshops. Because of her long-time 
involvement with the consortium’s activities, she has had multiple exposures and ample 
opportunities to talk with university instructors who attended the training with her. She passed on 
her insight gained from these conversations: 

Our main objective is to see students better off, and the Academy really helped with 
that— from curriculum and instruction alignment to improving classroom instruction. 
Having spoken with some of the university people at some of the meetings, I know they 
are adjusting some of their teacher education programs to reflect some of the things 
we’ve seen at the Academy. 

— Vignette ft 20 

A professor of science education in the North Central Region discussed how working with the 
regional consortium to provide professional development opportunities has improved capacity in 
training others, be that through workshops or more formal teacher preparation programs: 

I am the director of the Science Education Center and professor at a midwestem 
university. I have been involved with consortium projects since their inception back in 
1992. Initially, my involvement began through my participation on their advisory board. 
The consortium has also provided technical assistance the past several years in the 
implementation of the higher education Eisenhower grants I have received. For example, 
one of the workshops we do before our summer institute with the mentors is a workshop 
on facilitation. The last several years, the consortium would send us a staff of one or two 
to work with us in setting up a facilitation strategy workshop for these mentors. During 
the summer institute they would follow-up and come back with the whole group, mentors 
as well as peers, and introduce to the peers some of those same facilitation strategies we 
used in working with their mentors. As a result of the collaboration with the consortium, 
relationships were strengthened both with the university’s teacher education program and 
the state’s intermediate units. Working with the personnel from our regional consortium 
has certainly provided me with some unique experiences that enabled me to coordinate 
all of my efforts so much more effectively, as well as efficiently. I, just like the lead 
teachers, am learning immensely from what they do and I can use that in organizing and 
coordinating our entire efforts, not only in the grant-related work, but in my work as a 
pre-service educator here in our teacher education program. 

— Vignette ft21 

A university teacher-education instructor in the Southwest Region has also been involved with 
many consortium events and has passed the information she has gained to her student teachers. 
She credits the consortium training with giving her information about standards-based instruction 
and strategies for addressing multicultural issues. However, her most valued aspect has been a 
renewed enthusiasm for teaching. She explained: 

I have heard by word of mouth that my student teachers do indeed use things I have 
taught them from my consortium training when they are out there teaching, and that 
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pleases me. But I think what I get from the consortium, and can pass onto my students, is 
a definite energy boost — a revitalization of my love for what I do — and my curiosity 
stays alive and well. I would rather give these three things (energy, love of teaching, and 
curiosity) to my to-be-teachers than any knowledge or skills. The skills, they can always 
leam if they keep their curiosity and interest alive. 

— Vignette #22 

5. Impact on Informal Education Institutions 

The regional consortia also support institutions that provide education informally. As a result, 
these organizations are able to enhance their programming with what they have learned about 
standards and frameworks, and speak the same language as the formal educators they so often 
serve. An informal educator in the Mid-continent Region had this to say: 

As a performing arts organization, we would be considered an institution of informal 
education. A general sweeping statement could be made about people like myself — 
education directors — that we are performers who may not have a formal degree in 
education and so don’t know the education aspects. The training that the consortium has 
provided my colleagues and me has completely opened up a world that we were not 
aware of. I think a lot of us are getting a lot more savvy about who we chose to write 
curriculum for us, how we go about it, the potential of integrating curriculum, things like 
that. We are still, I would say, on the naive side to some extent, but it is very, very 
different now than it was ten years ago when I got into this business. We’re not taking a 
shotgun approach, but rather a very sequential one, where there is a reason for what we 
do and everything builds on something else; there is a lot more theory backing up our 
actions. As a result, the teachers we work with are more apt to implement the lessons 
because they know they are accomplishing some other, more tangible educational goals, 
in addition to exposing their students to the arts. 

— Vignette #23 

A participant attending a training offered by the Northwest Region consortium commented on the 
extent to which informal educators were excited about and engaged with the material being 
presented at the workshop. Furthermore, the training stimulated them to change their own 
practices: 

The people from informal science were the surprises — they were really positive about 
learning what national and state standards were, and how they could bring their programs 
in alignment with them. I had expected boredom and resistance from this group, but 
found them to be more excited than the formal educators! The small schools and the 
teachers from these schools have had at least one or two priceless experiences and will 
know now what they are missing. But what I will remember the most was watching 
informal science representatives become really enthusiastic about national and state 
standards, and actually eager to get home and change their programs accordingly. Several 
were people I saw again, and they had stories about improvements they had made and 
how much better their programs were. 

— Vignette #24 

6. Impact on District and States 

In addition to providing direct training to improve teacher practices and student outcomes, the 
consortia also targeted training and technical assistance to individuals and groups working at the 
district and state level in hopes of having a broader influence on the policies and practices that 
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support such improvements. The following interview segments illustrate the impacts of the 
consortia at the district and state levels. 

A district representative from the Far West Region described how their district was able to build 
upon the training provided by the consortium to align their curriculum with the standards and 
track student performance: 

Our district is currently modifying its report cards to be standards-based. The consortium 
has supported our district by providing Learning from Assessment workshops. The 
Learning from Assessment module provided a framework to use in aligning the standards 
with the curriculum and then thinking about how we assess progress on the standards. 
Starting with this framework, the teachers began working on a matrix to see where their 
curriculum and the textbooks they were adopting and the supplemental materials they 
were using were meeting the standards. As a result, the teachers feel like they have a plan 
for meeting the standards. Keeping the goals of the standards in mind, we’re looking at 
our students to see how they’re performing as we’re doing our lessons and looking at 
what that is telling us so we can reach our goal of meeting the grade-level content 
standards. As a result, there is more purposeful teaching happening, and it’s meeting a 
wider variety of student ability needs. 

— Vignette # 25 

When other regional providers participate in consortium training activities, along with staff from 
the districts they serve, the positive effects are evident throughout the education system. In the 
Northeast and Islands Region, a representative from one such provider organization described 
how their ability to work effectively with districts was enhanced by the ways in which the 
channels of communication opened up among key players in the education process: 

As a participant in the Using Data workshop, it made me think about data and how I use 
data, or don’t use data. It caused us as an organization to start asking some questions 
about data leading us, versus us trying to figure out what to do. So, it caused us to have a 
little discussion about our organization, which is always healthy. And about how we 
should be asking some questions maybe a little differently than we were, and use data to 
drive some of our decisions. And it really did increase the quality of conversations we 
were having with our districts. Quite frankly, it gave us, as a regional service provider, a 
doorway in to help them with those conversations. So the workshop was kind of like a 
can opener. We were able to help the districts focus their attention on standards-based 
curricula, after looking at their standardized state assessment scores and trying to 
determine what was going on. Their questions often led to the conclusion that the state 
assessment isn’t sufficient to know everything they want to know about how their 
students are doing. We asked them to look at other tests they’ve taken, look at teacher 
kinds of things, and look at student work. The workshop increased the quality of the 
conversations the school districts were having, teacher to teacher, administrators with 
teachers, parents and teachers, about really improving practice and improving the content 
focus for kids. Looking at their data and experiencing that workshop has certainly 
influenced the way some districts are now approaching curriculum selection... they’ve 
gone beyond the “five-minute-thumb-through-the-books” method to a whole piloting 
process, based on looking at the data. 

— Vignette #26 

Additionally, it is believed that changes brought about by technical assistance at the district level 
may also “flow upward” to influence state-level policy makers. Examples of consortium activities 
included state-level meetings that often targeted state-specific issues or systemic initiatives. 
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Consortia members have assisted state education agencies in designing state-level student 
proficiency exams and in aligning state exams with national standards. They have also assisted 
states in building state-level capacities to understand and disseminate information about TIMSS 
reports and to provide professional development regarding strategies to help close the 
achievement gap between Caucasian students and students of color. 

In the Appalachia Region, a representative from a state department of education commented on 
her involvement with her regional consortium in providing professional development 
opportunities for educators on classroom equity issues through a regional equity conference. The 
2000 Equity Conference was held in collaboration with a state education agency: 

I was involved as a conference-planning coordinator. I collaborated with the consortium 
mainly in planning technical assistance activities for educators of under-represented 
students in our state. One of the presenters at the equity conference provided strategies for 
working to close the achievement gaps with minority students. Our division used her 
strategies to develop a policy that focused on student achievement. This policy is being 
piloted in six districts that have 72 percent of the minority student population in the state. 
The policy requires that the strategies be used in the classroom to show achievement. There 
are two checkpoints for improvement, FY 2002 and FY 2004. We hope to see a difference 
then in the 4-6 grades in terms of closing the achievement gap. Following the pilot, this 
policy will be available statewide. Equity is an extremely important topic. By collaborating 
with the consortium, I have access to resources that assist me in my work and assist our 
division in providing services to schools and districts with a majority of at-risk students. 

— Vignette #27 

In the Far West Region, a client utilized the consortium to provide professional development around 
standards-based education for all of the tribal schools in the state: 

When the state accountability test was adopted in our state, teams from the eight tribal 
schools came together to try to align our curriculum to the standards. At the point we 
started, you couldn’t recognize any similarities in the curriculum from one school to 
another. Even though each school needed to retain their individuality and how they 
delivered their education, we had to adjust in order to standardize our curriculum across the 
Indian community. That’s when the consortium was brought in to do professional 
development, because we’d get stuck wondering, “How do we do this?” Last fall we had 
two professional development programs for all teachers out at all eight schools on how to 
do this. Everybody started recognizing, that “Yeah, this will work. It may take five years, 
but we’re going to see a little bit of difference every year!” 

— Vignette #28 

7. Impact on Training for Trainers 

In addition to sponsoring and providing direct training and technical assistance to teachers and 
other educators, the regional consortia also foster systemic reform through the use of a train-the- 
trainers model of professional development. Participants in the train-the-trainer sessions include 
district and state-level staff, representatives from institutions of higher education and professional 
organizations, and teachers. 
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• District and state-level trainers served. A long-time participant in the Appalachia Region’s 
consortium-sponsored events for trainers commented on the consistency with which the material 
helped the trainers to enhance their understanding of standards-based education and the strategies 
for bringing such a system into being: 

I have been involved as a team leader and trainer for several years, and every time a team 
leader or train-the-trainer meeting took place, we learned about new ways we could help 
teachers leam more about the national and state standards and their curriculum. When 
teachers attended our consortium training workshops, they were provided with 
instructional practices to use in their classrooms that were aligned with national and state 
standards. 

— Vignette #29 

Ln the Appalachia Region, a director of mathematics, science, and technology discussed how the 
consortium’s training-of-trainers model supported the district in developing a new curriculum: 

Our school district has received tremendous support from the consortium over the past 
three years as an intensive school site. When we were looking to adopt a new, standards- 
based math curriculum, the consortium sent us to workshops that gave us a first-hand 
look at several math curriculum materials, and then prepared a cohort of teachers and 
staff developers in our district to train the rest of the teaching staff in using and 
evaluating the Investigations curriculum we adopted. Our staff developers and trainers 
brought back ideas about what needed to be done in the classroom to effectively get 
students to understand mathematics. We’re trying to get away from algorithms and talk 
more about mathematics in depth, but we’re finding that our teachers really don’t have an 
in-depth background in mathematics. So we’re providing them with professional 
development to understand the math content and what kids think about when they talk 
about mathematics. Now, what I’m seeing in classrooms is students working together in 
groups to solve problems; whereas, a year or so ago I saw more students sitting passively 
and just listening. 

— Vignette #30 

District trainers in the Southeast Region also commented on the role their regional consortium 
played in helping them to map out strategies for supporting teacher leaders: 

The consortium has been providing us with several different services related to the 
development of teacher leaders in the district. I think that what the consortium did in its 
work with us was to clarify important roles for teacher leaders and to provide some 
models for what reform should look like at the classroom level. I would have to say that 
they helped us to formulate the rationale and support for a site-based professional 
development model that focuses on coaching teachers at the school site and developing a 
deeper capacity within the school building to sustain reform. Not only did the consortium 
staff help me personally and other staff in the district with the formulation of that 
strategy, but they affirmed that this strategy has credibility beyond just our district. 

— Vignette #31 

Consortium-sponsored training-of-trainers was also cited as a means of enhancing the knowledge 
of state-level trainers that ultimately led to improvements in the quality of training provided to 
others. In the Northeast and Islands Region, a state-level trainer explained: 
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I work with teacher leaders in schools across the state who are responsible for designing 
ongoing professional development for their colleagues in order to improve their math 
program. Last spring, I attended a two-day workshop on Case Studies so that I could 
share that approach to ongoing professional development with the teacher leaders. The 
workshop was fabulous. It exceeded my expectations. As a participant, I was impressed 
by simultaneously receiving training as a participant in the process and being shown how 
to roll it out and to share it with others. I think that the facilitators were so skillful in 
making explicit the different roles and ways to participate. 

— Vignette #32 

A state science coordinator in the Pacific Region described how involvement in the consortium 
increases confidence and empowers state-level trainers to do their work more effectively, 
particularly when it comes to working with the people in the trenches: 

I’ve participated in consortium activities on a number of levels, including the 
development of regional standards for science, planning and providing professional 
development for teachers, providing math/science leadership, facilitating the Visions and 
Dreams program, and building networking and collaborative relationships within and 
between different agencies. My participation in the consortium has been a tremendous 
learning experience for me. I’m better prepared, more confident, and better able to do the 
work that I do. I’ve become more knowledgeable about how to facilitate the development 
of local science standards. I feel empowered to plan and conduct professional 
development aligned with our curriculum. I’ve been positively challenged to look at 
assessment differently. I’ve taken personal responsibility to guide and facilitate 
discussion in using multiple assessments in the classrooms. Collaboration with the 
consortium has also forced me to be more resourceful, to more creatively use what is 
available. I’ve become more proactive in the design and implementation of professional 
development and other training activities. I’m very excited about the possibilities that 
have opened up in helping me to meaningfully connect with the top people in 
education — the teachers. I know that if I can provide them with the tools and strategies, 
they can move and motivate students! 

— Vignette #33 

Still another state-level trainer, a mathematics content specialist from the Pacific Region, was 
able to talk about the way in which helping trainers improve their own professional practice leads 
to changes in teacher practice: 

To me, the most significant impact of the consortium is that it has helped teachers to 
change their teaching strategies from predominantly teacher-centered instruction to 
student-centered instruction. I have seen significant changes in teaching practices 
throughout our schools. I’m also looking at classrooms differently as a result of my 
involvement with the consortium. As a result, I’m able to help teachers think differently 
about teaching strategies, questioning strategies, grouping strategies, and classroom 
activities. For example, teachers are using more hands-on activities and manipulatives 
with their students. They are trying out different questioning strategies, using questions 
that require the students to do more thinking. Teachers are now more concerned about 
how students came up with the right answer, and not just the answer alone. Teachers are 
also incorporating authentic assessment and are changing their report cards in ways they 
think give a clearer picture of the students’ learning. It’s not a finished process. We still 
face many challenges. I know they and I still have a lot to learn, but I believe we are on 
the right path. 

— Vignette #34 
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• Trainers from institutions of higher education served. A mathematics coordinator for 
a technical assistance center located at an institution of higher education also serves as a 
trainer for the consortium in the Appalachia Region. She had this to say regarding the impact 
of her own training in allowing her to assist teachers in assessing whether their curriculum 
and instructional practices were aligned with standards: 

I went to the Learning from Assessment professional development opportunity and learned 
a great deal. The training was using a model for determining if a curriculum is aligned 
with national or state standards. This fits really well into the mission in our state that 
teachers need to be able to assess if their curriculum is aligned with the standards. I have 
used this model in the professional development sessions I have conducted and have 
probably trained more than 200 educators on this process. I not only provided information 
to educators on aligning their curriculum with the standards, but also in determining if 
their instructional practices were aligned as well. I believe the materials I received from 
the consortium have been very helpful in providing teachers with an opportunity to reflect 
on their assessment strategies and determine if they are aligned with the standards. The 
consortium has done a wonderful job of making sure that their training is aligned with the 
goals of our state. We get the national perspective in terms of the standards and then adapt 
it to our goals. 

— Vignette #35 

This same trainer went on to comment on the role of the consortium in the Appalachia Region in 
helping her to stay abreast of innovative models and strategies for providing professional 
development: 

The consortium has provided me with new, innovative models for providing professional 
development. Through my work with the consortium — as a trainer and team leader for 
the Regional Training Project — I have experienced first-hand how to provide professional 
development that fits both the needs of my organization and the needs of our state. This 
has been truly valuable to me to continually improve on how I provide professional 
development by modeling what I receive from the consortium. The professional 
development was useful, innovative — yet practical — and aligned with the mission of our 
center and the state. It allowed me an opportunity to gain knowledge that I could then 
share with others. When I provide professional development, I am using the innovative 
models and strategies that I learned from the consortium. This helps to support my efforts 
at the center. 

— Vignette #36 

• Trainers from professional associations served. The regional consortia also work with 
professional organizations on training issues. The director of a professional organization in the 
North Central Region discussed how the train-the-trainer model was particularly useful in 
enhancing the knowledge of someone who is responsible for assisting teachers in so many 
different content areas, and the ways in which this empowered those teachers to assist others: 

The collaboration with the consortium arose out of a grant that we were given to assist 
public school teachers in the seven school corporations that I work with in the use of 
technology in advancing mathematics and science. Part of my job is to help the people 
that I represent to grow and develop more professionally. I am not an expert in all 
curricular areas, and so sometimes it becomes very difficult for me to know what we 
should be offering or what help we should provide. Being involved with the consortium 
and being involved with this collaboration added significantly to my understanding of 



things I didn’t know about until I had gone through some of the trainings and in-services 
that the consortium staff provided. It made it much easier to design and offer professional 
development opportunities for the 1,600 teachers that I represent. I think that probably 
the strongest outcome from all of this is that the teachers that we had involved in this 
collaboration have now gone back to the school corporations and into their buildings and 
trained other teachers. It has been very much a domino effect. Another outcome that I can 
point out to is how it has fired up some of the teachers that I represent to go ahead and 
move in directions they hadn’t moved before, to try things that they hadn’t tried before. I 
think that the most positive outcome was that it really leveraged and increased the quality 
of instruction that is offered to the children here. 

— Vignette #37 

• Teachers learn to be trainers. Regional consortia also train teachers to train other teachers. 
A high school science teacher in the West Region described how preparing teachers to be trainers 
can help an entire district: 

The consortium has supported our district by providing Learning from Assessment 
workshops, with the goal of providing teachers with the tools they need to align their 
math curriculum and assessment with the standards. I’ve participated in the LFA training- 
of-trainer workshops. Since then, I’ve been helping my district to hold workshops for 
other teachers so that that they can understand exactly what our students are going to be 
tested on. Now when I hold professional development workshops, I can make sure that 
the teachers in my district understand how the test questions are going to be written, and 
they can go back to their classroom and make sure that they’re teaching it correctly. 

— Vignette #38 

A state science coordinator in the Far West Region talked about the way in which an initial 
training for a relatively small cadre of teachers can lead to manifold returns in the total number of 
teachers that are prepared as trainers: 

The consortium helps us with our Phase II elementary in-service, where we cycle through 
on a three-year cycle to each grade level, doing grades K-l-2-3-4-5-6, and repeating the 
cycle. All 40 of our state’s districts participate in this in-service, where the constructivist 
strategies being taught help students to develop an understanding of science process skills 
as defined by the intended learning outcomes in our state. It’s the hands-on, minds-on 
model where we try to connect the learning in science to an experiential base. This in- 
service program trains a cadre of about 100 to 110 teachers each summer in an intensive 
workshop that the consortium has been supporting for the past six years. Then, using state 
monies, we send the teachers back to their districts, where they train additional teachers. 
It’s a model where we build the capacity of our cadres in the summer through intensive 
training. About 100 teachers that are trained in the summer typically turns into 3,000 
trained teachers by the end of the school year. 

— Vignette #39 

Another teacher, this one from the Southeast Region, also commented on the ways in which the 
transference of skills and knowledge from one teacher to another builds upon itself: 

From the beginning, my participation with our regional consortium has strengthened me 
as a classroom teacher in helping me to align my curriculum and my classroom 
instruction with the standards. But I’ve also been able to take that to a much broader 
audience. I worked for an entire week with one of our state universities (through a 
National Science Foundation grant our school district received) to provide a series of 
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workshops for secondary science teachers on a standards-based instructional model. I was 
able to deliver those sessions using consortium materials, using a workshop that had 
already been proven successful with teachers, and then deliver it again to another group 
of teachers, with the intent of their changing their classroom practices. 

— Vignette #40 

In summary, as the education director for a special science program in the Southeast Region 
noted, the train-the-trainer model employed by the regional consortia plays a significant role in 
preparing trainers at all levels: 

It was my training ground. It was the way I was exposed to nearly every aspect of what I 
was going to be expected to do. It was really vital. The most significant thing I’ll always 
be grateful to the consortium for is my professional development in being able to 
understand my job and help me work more effectively as a trainer. It exceeded anything I 
could ever expect because I didn’t know that I would be so well prepared to do this 
particular job as a result of my being in the Academy. 

— Vignette #41 

8. Effective Technical Assistance and Training 

This final vignette illustrates the wide-ranging impacts of the technical assistance and training 
that is provided by the Eisenhower Regional Consortia. It demonstrates the consortia’s capacity to 
enhance teachers’ ability to align instruction with state standards, to deliver effective instructional 
strategies that motivate students to learn deeply the material that is presented, and to become 
instructional leaders for assisting other teachers through the consortia training. The vignette also 
shows how the services that the consortia provide have a rippling effect — which affects education 
in deeper, more systemic ways that go beyond the initial training and technical services that were 
delivered. An award-winning mathematics teacher in the Southwest Region explained: 

The consortium worked with a group of Master Teachers in our state. Some of them are 
presidential awardees from past years and others are just known as leaders in their school 
or in their district. The training certainly helped me to improve the way that I teach. It 
gave me a lot more ways and things to incorporate into my daily routine as a teacher. I’ve 
been teaching now for 17 years and when I started out, I was very book-oriented. I didn’t 
care what the curriculum said — if I was on page 30 one day, I was on page 3 1 the next 
day. Because of my training, I’ve come to understand that aligning instruction with 
standards is the whole point. I don’t go from one topic to the next just because it’s in the 
book that way — instead, I do what makes sense to the kids. 

Additionally, the training has given me a lot of tools to be able to appropriately pace my 
kids. Kids get through the curriculum and, being a Title I school, we have a lot of at-risk 
kids, so that’s a major accomplishment. Now, students are working at a much higher 
level whenever they have all those manipulatives around to work with; it helps with their 
thinking skills. And — this is part of what I learned from my work with the consortium — 
memorizing is nice, but what you really want to do is teach students thinking skills and 
help them understand the process they’re going through. Then, students understand why 
the math works like that. Instead of me saying, “Don’t worry about that, just follow these 
three steps and solve the problem,” they understand the three steps that they are going 
through and the reason why. Additionally, I think student engagement is most important 
because if they are not interested, they sure are not going to do anything. I mean, that’s 
why you improve your instruction, or make the curriculum better, or assess differently — 
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to make the whole thing more interesting. And that way, students want to leam and they 
do. 

I’m trying to influence others through my leadership at the school, my district, and in the 
workshops I do for the state. I try to show other teachers how to be a good teacher and 
tell them why it’s so important. I make them copies of my materials and take time to 
show them things the kids will like. So, I’m telling other teachers, and my principal, and 
the parents about the importance of improving the way we teach. There are at least 50 
teachers in this state that are, or were, in the Master Teachers program. Each of them 
helps a few more teachers. We are changing what we do; at the same time we are letting 
more people know what we need to do to be good teachers and why. 



The consortium has made more people know how important it is to get students to want 
to leam, to “teach curiosity” like the consortium staff member always said. The 
consortium has also made a difference in attitudes. For instance, my principal has been 
very supportive. She is a former math teacher so she’s behind me. But other people’s 
principals, who are hearing some of what we are learning, are more supportive of 
teachers who want to upgrade the way they teach. And, instead of refusing time off for 
meetings and things, they encourage teachers to go and make it easier for them to do so. 
There’s no problem about release time for any of the Masters Teachers activities. My 
district is very supportive. And that might not have been true if it weren’t for the 
consortium. 

— Vignette #42 



B. Dissemination 

The Eisenhower Network offers a unique service by providing a point of contact where educators can 
leam about and access regional and national resources for mathematics and science education. 
Dissemination activities are designed to improve participant practice and student engagement and 
performance by providing resources to improve curriculum and instruction, such as information about 
exemplary and promising practices in mathematics and science education. 

Earlier Network evaluation reports have relied upon quantitative data to document these activities 
and their impacts. This section presents vignettes created from in-depth client interviews to 
further illustrate the scope and nature of the dissemination activities and the impacts these 
activities have had on our clients. It is important to note that the client interviews were not 
originally designed to elicit outcomes of dissemination activities; however, it was possible to 
glean relevant information from clients’ comments about the roles the consortia and 
clearinghouse play, and the unique benefits the Networks provides. 

The following vignettes describe how clients value both the quality and utility of products developed 
by the Network or supported in collaboration with other organizations. These vignettes show how 
clients see these dissemination activities as providing unique resources, supporting product 
development, enhancing product dissemination, and changing practice. 

1. Unique Resources Provided 

A statewide systemic initiative project manager in the Northeast and Islands Region shared how 
products impact policy makers: 
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The consortium is a great source of information funneled from the national level and a 
point-of-contact for the regional level. The consortium has provided us with information 
and professional development on TIMSS, effective professional development strategies, 
and strategies for using data. Some of those were unique materials, and if the consortium 
hadn't done it, who's to say that it ever would've been done? It’s been great to have a 
nearby partner creating products and services that we can use here, particularly for issues 
or topics that, frankly, are important, but peripheral enough that we don’t get to it; where 
our state has some difficulty doing it, and enough of the states in the area all recognize 
that we’re having trouble, but we know it’s important. Somebody needs to do it, and it’s 
nice to have that regional entity that can respond to that. The consortium has done a great 
job at creating the program, tool, or product that fills a need many states have identified, 
therefore preventing duplication. 

— Vignette # 43 

A district program consultant/content specialist in the Pacific Region shared how the consortium 

provided much needed materials that added value: 

Schools look to me to provide training on content, pedagogy, and current educational 
issues. It’s not always possible to provide the needed materials and information, or at 
least it hasn’t been in the past. Fortunately, because of our involvement with the 
consortium, local initiatives and activities related to standards-based education have been 
carried out. This is critical because the local resources are insufficient to support these 
activities. The consortium has allowed us to provide materials from several different 
organizations. These materials identify effective teaching practices, support quality 
professional development, and provide ideas and material for standards-aligned curricula. 
— Vignette #44 

A technology coordinator for a public high school and a coordinator for an ENC Access Center in 

the Appalachia Region shared how high school students have used Network-developed materials: 

Our high school was selected as a site to house an ENC Access Center. . . We not only 
offer the resources to educators, but our students have access to ENC resources as well. 
These high school students are doing research that most college professors would only 
dream of doing. A few years ago, I was asked to work on a landfill issue for our area. 
When approached, I said that I wanted my students to work on it. Two of them attended 
every meeting and studied the problem using resources from ENC. They recommended 
if we recycled plastics, the landfill space would decrease by about 25 percent. A year 
later, a major organization came out with a very similar finding. We were ahead of them 
by one year! 

— Vignette #45 

A central district director who has worked with ENC shared how resources have informed 

teachers: 

I look at ENC resources mainly for national and science standards and curriculum 
materials. ENC makes information available to science educators quickly. The 
information is condensed, and the magazines and publications are current. Once teachers 
know that ENC is here, they are a resource for a teacher in the classroom, especially if 
that teacher has online access. The information is right, available, sorted, and at one site. 
Teachers do not have time to sort on their own. And, think about teachers who do not 
have connections with computers. ENC’s Focus and other publications connect teachers 
without computers. 
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— Vignette #46 

2. Product Development Supported 

A state science consultant in the North Central Region shared how the consortium assisted with 
the design and development of an equity toolkit: 

One of the projects I worked on recently with the consortium was the Connecting With 
the Learning Equity Toolkit. The consortium provided technical assistance to the state 
and to a lot of our professional developers in helping with the development of 
professional development for the toolkit. They also helped with the design and 
production of the toolkit. Most recently, they helped with the evaluation of the toolkit 
with teachers in the Midwest. If it had not been for the consortium, we would not have 
been able to get the toolkits printed, and we wouldn’t have had quality products if it had 
not been for the services and support of the consortium. By us being able to produce a 
product of such high quality, we were able to generate a lot of interest and participation 
in workshops related to the toolkit. We still get calls from all over the United States and 
people who want to use it in their school districts. Much of that was done because of the 
leveraging of the work done through the consortium. 

— Vignette #47 

A mathematics coordinator for a technical assistance center located at an institute of higher 
education in the Appalachia Region shared how the consortium supported the development of a 
product with the center: 

The consortium has supported the development of materials that address specific needs of our 
state and region. I collaborated with the consortium in developing a graphing calculator 
product. This product was essential in terms of equity for all students. I have used the 
graphing calculator and K-8 building blocks materials extensively. Not only does the 
consortium provide materials that can be used across the region, but also within our region 
they have provided a consistency in terms of the training packages that allows for a statewide 
training mechanism. I believe that the consistency factor is very important there. 

— Vignette #48 

3. Product Dissemination Enhanced 

A secondary science teacher specialist with a state department of education in the Far West 
Region shared how the consortium assisted with the dissemination of their product to reach 
teachers statewide: 

The state office of education recently developed an Earth Systems high school science 
course. The consortium provided the technical assistance to put all of the lesson plans on 
the Web, so they are available to teachers statewide. The course contains inquiry- 
oriented, activity-based lessons. The consortium put our Earth Systems methods course 
online to help teachers implement the lesson plans. The online lessons model 
instructional practices espoused by the National Science Standards. Because there are 
very few lesson plans and no textbooks aligned directly with the state core, teachers were 
scrambling for resources. I’ve had many teachers express appreciation for the fact that 
there were Earth Systems lesson plans tied directly to the state core that were easily 
accessible. If the lesson plans weren’t put on the Web, the teachers wouldn’t have 
accessed them. 

— Vignette #49 
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A client associated with a non-profit organization in the Mid-Atlantic Region shared how the 
consortium assisted with the dissemination of a professional development product that allowed them to 
reach a greater number of teachers: 

By networking with the consortium, my organization was able to expand the 
dissemination of a professional development video series based on national mathematics 
and science standards. The videos are broadcast on a television channel that can be 
reached by half the schools in the country. Print and Web-based materials that 
accompany the videos encourage teachers to reflect on their own practice. The video 
classrooms represent broad geographical and racial populations. As a member of the 
consortium’s regional board, I was able to make the materials available to all of the board 
members. One of the most significant outcomes of this networking with the consortium 
has been the engagement of teachers in workshops in 200 sites across the country. The 
number of teachers we know we are reaching increases every semester. The 
consortium’s strength in this is its ability to promote interstate activities and share 
materials amongst states by providing an opportunity for individuals to network, and their 
unique contribution is the advocacy of the consortium for the teachers to use the 
materials. A number of teachers would never have known about them without the 
consortium’s involvement. The end result is that a larger number of teachers are more 
aware of standards-based education. 

— Vignette #50 

A professional development provider/regional service provider in the North Central Region 
discussed how resources were used in an equity workshop: 

We have used a number of the consortium’s resources. We just finished an equity 
workshop, and they have an equity toolkit which is excellent. We have used bits and 
pieces out of that for some of our presentations. When people call with equity questions, 
we suggest that as a resource. They have something called Science T.R.E.E., which 
allows teachers to design their own science units. We’ve shared that with a number of 
science people who are science staff developers for their school districts, usually the 
larger districts, so they can inservice their teachers on how to use this. They do have 
wonderful resources that are available; so we have taken advantage of those. The 
products they produce are very valuable. 

— Vignette #51 

4. Products Change Practice 

A science coordinator with a public school district in the Far West Region described how 
materials from the consortium enhanced her effectiveness in serving teachers: 

The publications that come out from the consortium are valuable. They come to me, and I 
pass them out to teachers who might need various things. They send me a catalog of 
what's available through the consortium; they also send me booklets once in a while, and 
ENC stuff, math and science, is valuable. It lets me know where I can go to get things, or 
what other people are doing, or where the departments are that we can connect into, so I 
find that very useful. I think that it's been a good resource for teachers, like it has been 
for me. They use it primarily in the same way I would. The information from the 
consortium has helped me do my work more effectively. It helps me to be knowledgeable 
about what's going on in other states and what are best practices there, and those types of 
things. It helps me to do my job better. Anything that I can draw upon, that I can pass on 




30 



43 



to my teachers, helps this job be better. That's my whole purpose, to be of service to my 
teachers, and so the more that I know and understand and know where to get things, or 
how to do it, then I can pass that to them, and that consequently affects our students. 

— Vignette #52 

A university professor in the Southwest Region shared how a Toolkit for Alternative Assessment 
disseminated by the consortia enhanced his teaching pre -service teachers: 

One of the things we received was the Toolkit for Alternative Assessment, and I've 
grown so attached to it that I can't leave home without it. It’s the best of any materials 
I've ever gotten in connection with anything. By now, I've given four workshops on 
alternative assessment, or on assessment period, and I’ve always given the toolkit. If 
others just got the toolkit without my teaching, they would still be way ahead of everyone 
else. I really can't say enough about the toolkit. I did not ever expect assessment to be an 
area of interest or expertise, but it may be my favorite workshop to conduct. 

— Vignette #53 

An educational program materials specialist for a NSF-funded technical assistance provider in the 
Appalachia Region shared how becoming an ENC Access Center improved their capabilities to 
provide standards-based professional development: 

The consortium is a link to national-level products on mathematics and science 
education. Serving as an ENC Access Center has allowed us access to quality materials 
for my use and in my training sessions with educators. When I do my training sessions 
for ENC Access Center, I pull instructional and assessment strategies from the various 
ENC materials and I use them as demonstrations in the training, showing educators how 
they can use these materials in their classroom and where they can access the information 
from ENC. This type of strategy works well in our state. Our state standards are aligned 
with the national standards. ENC materials are aligned with the national standards. 
Therefore, I can use nationally focused products that are aligned with my state and 
schools. Through their role in dissemination, the consortium has raised the level of 
awareness of seasoned teachers. The consortium provides a lot of materials and 
resources to all teachers, but I feel that they also provide an opportunity for seasoned 
teachers to get up-to-date on the latest developments in mathematics and science 
education. 

— Vignette #54 

A regional service provider in the Northeast and Islands Region discussed how the consortium 
provided resources that enhanced the organization’s ability to deliver professional development: 

The consortium has been a great resource to me. They've got a lot of tools and resources 
that they help me become aware of and use. When there are products that come out from 
the other Eisenhower Consortia, [the director] is really good at making those available. 
Specifically, the resources and expertise available at the consortium on professional 
development and using data have really enhanced the work of my whole organization. I 
attended a session on professional development at a conference, early on when the 
professional development book, Designing Professional Development for Teachers of 
Mathematics and Science , was in draft form. Learning about that book and how it can be 
used in our organization to design effective professional development was a learning 
experience for me, which then I was able to take back to our organization. We were able 
to get the book when it came out in print, and we were able as a staff to design some 
internal, professional development around the book. And now, I mean, that’s our guide. 
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We use that book in all the work we do and that goes back to being introduced to that 
book through the consortium. And the same thing would be true with the consortium’s 
data book. 

— Vignette #55 

A state science supervisor in the Northeast and Islands Region explained how the consortium 
provided information that helped influence state policy around standards and curriculum 
frameworks: 

In the eight years that I've collaborated with the consortium, they have assisted us in ways 
that have had a tremendous impact on the services we provide to the school districts in our 
state. Because of the consortium, we have been able to provide our state with print 
resources and comprehensive professional development that we wouldn't have been able to 
otherwise. The consortium provided all the districts with copies of the Professional 
Development book, and the Equity Manual that our consortium state team created the year 
before, as well as copies of ENC Equity CD, and all of the rich resources that come out of 
the Eisenhower Clearinghouse. The consortium provided support that enabled us to create 
addenda to our state frameworks and distribute them to all schools in the state. If it wasn't 
for the small amount of money that the consortium provided, in terms of helping us put 
those addenda together, the districts would not have had any context for interpreting those 
standards. More significantly, it helped us, because of those addenda, identify needs in the 
pre-service preparation of teachers, and as a result of that and lots of other activities, we 
have new policies now in the state that reflect standards that have been identified in our 
framework, but also standards that have been identified in those addenda. 

— Vignette #56 



C. Collaboration and Networking 

Collaboration and network building is another major service area addressed by activities of the 
Eisenhower Regional Consortia and Clearinghouse. This service is intended to encourage 
communication, collaboration, and shared activities among various client groups in order to 
enhance working relationships and build professional alliances. Activities can include referrals, 
sponsoring meetings and conferences, providing resource information that enhances 
communication and collaboration, supporting communication networks among working groups, 
and providing seed money for collaborative efforts. 

The Eisenhower Network’s collaboration and networking activities were designed to improve 
participants’ practice and student performance through increased access to resources, increased 
coordination of services, leveraging of resources, and strengthened relationships. Earlier Network 
evaluation reports have relied upon quantitative data to document these activities and their 
impacts. This section presents vignettes taken from in-depth client interviews to further illustrate 
the scope and nature of the collaboration and networking activities and the impact these activities 
have had on our clients. 

The client interview data are found to be consistent with the larger-scale survey data reported 
previously. In response to a series of closed-ended interview questions, approximately nine out 
of ten collaborators reported their collaboration with the Network strengthened relationships, 
helped them to work more effectively, increased access to resources, and/or leveraged resources 
for greater impact. The same number reported this collaboration met or exceeded their 
expectations. Approximately eight out of ten reported their collaboration with the Network 
increased service coordination and/or provided benefits not otherwise afforded. The Network’s 
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collaborative work had lesser, though still notable, impact on policy: six out of ten collaborators 
reported that their collaboration with the Network informed policy decisions. 

Illustrations of the scope and nature of the Network’s direct collaboration with clients can be 
found in comments recently offered by an associate director of a technical assistance center from 
the Pacific Region: 

I’ve been actively engaged as a collaborating partner in the consortium in a variety of 
ways: in planning and product development; in collecting and analyzing impact data; in 
planning and delivery of professional development for the Consortium Leadership team 
and through work on Pacific Standards. The key theme I see running throughout the 
work is one of networking and collaboration. Our collaboration has grown in a way that 
has allowed us to successfully navigate a rocky start to achieve true collaboration at all 
levels. We have been able to work through the rough parts of organization functions, 
priorities, territories - the politics - to achieve a successful collaboration. I believe that 
our collaboration increased coordination of direct services to teachers in each school 
system we worked with. Better collaboration and networking meant increased access to 
resources, in the sense that we were able to better leverage available resources to achieve 
maximum impact. Perhaps best of all, through enabling staff to more clearly 
communicate the results of what works, we have informed policy at the top levels. For 
the first time in over 20 years of working in the region, we have a common vision and 
sustained efforts at achieving that vision. It’s an exciting time. 

— Vignette #57 

In addition to direct collaboration with clients, the consortia also promote collaboration and 
network building by supporting the work of other providers. The value of this support is 
illustrated by the comments of a mathematics/science coordinator from the Mid-continent 
Region: 



We had direct resource support from the consortium that allowed us to offer workshops 
to teachers and to disseminate information around standards-based education. We were 
able to use this support to bring teachers together to share ideas, give them materials they 
could actually take and use in the classroom, and even follow up with them to see how 
things were going and how they were using the materials. The biggest impact was getting 
teachers to use more hands-on activities in math and science instead of just having the 
kids do algorithms or a cookbook lab. We were able to move them to the next step. We 
would not have been able to do that at the time and now we definitely wouldn’t be able to 
with the budget shortfall that’s coming. 

Our district is also fairly large and we have all these different programs going on with 
really no coordination of that information. Increased coordination was one of the things 
that we were able to do with this support. For example, we have PRISM kits that are 
designed by teachers for use in math and science. The resources from the consortium 
allowed us to help set up a process for the dissemination of those kits. Along these lines, 
it really helped us to be able to say, “look we’ve been able to strengthen the knowledge 
base of teachers and supply materials with the help of the consortium.” 

— Vignette #58 

During the course of reviewing consortia client interview responses, certain response patterns 
began to emerge. Responses to collaboration and networking probes seemed to focus in the areas 
of relationships, reform, and resources. Indeed, when collaborators were asked to identify the 
most significant outcome of their collaboration with the Network, the most frequent responses 
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were categorized around the following three impacts: strengthened relationships, reform 
synergized, and increased access to resources. 

1. Relationships Strengthened 

As stated before, many of the consortia activities were designed to improve participants’ practice 
through strengthened relationships. During the interviews, many clients mentioned that these 
relationships consequently made it easier for stakeholders to share ideas and expertise and for 
teachers to increase their leadership skills. 

A science supervisor from the Appalachia Region described what she saw as an outcome of the 
consortium’s work: 

Teachers having the opportunity to network with other teachers when they attend training 
workshops sponsored by the consortium has been the most significant outcome of the work the 
consortium has done. When teachers get together, they will often realize that they have access 
to quite a bit in their own school district. I believe that teachers are pretty isolated in their 
classrooms, and when they come to these training workshops, they get an opportunity to share 
so much with one another. During the workshops that I’ve conducted, they will often 
exchange e-mails and phone numbers with one another, and we encourage them to do this. 

— Vignette #59 

An administrator for a public school system from the Southeast Region also shared comments on 
how relationships have been strengthened in their region: 

Involvement with the consortium has strengthened relationships at several different levels. It 
strengthened relationships between the leadership in the consortium and the leadership of the 
reform effort in the district. It strengthened relationships between teacher leaders and other 
teachers in the district. It strengthened relationships between teacher leaders and school 
principals. It built an infrastructure that allowed us to move forward with the reform agenda, 
and it definitely increased the leadership capacity of our teachers, and it also helped afford 
those teachers to get a deeper formulation of what the reform math and science classroom 
should look like. 

— Vignette #60 

The following comments from a mathematics and science administrator in the Far West Region 
illustrate that relationships established through the Network’s collaboration and networking 
activities are effective at overcoming the isolation felt by stakeholders and yielding successful 
results for teachers and students: 

For six years, I've been trying to get all eight schools to come together and do a 
community science fair. These schools are all different kinds of schools. There are BIA, 
Catholic mission, charter, tribal, and our public school district, and they have not 
communicated together at all in the 30 years that I've been involved in the community. 

And what the Tribal Coalition did was come in and provide a forum and use the 
consortium to bring representatives from each of the eight schools together, and we met 
every month and sometimes twice a month. That was one of the very positive things that 
occurred, bringing these people together to talk, every month, and finally working 
through their isolation. The worst enemy of tribal teachers is isolation, and so that's 
where I've seen the very best from the consortium! As a result of bringing people 
together, it's helped people to realize that everybody has the same problems, and to start 
working together and talking together. We later involved the teachers from these schools 
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in three-day, outdoor workshop where they did hands-on science activities. And I saw 
more light bulbs go on, because most teachers have been brought up in the traditional 
way where you teach in a classroom, you don’t go outdoors where you don’t have control 
over the kids. And whereas earlier I had had such a difficult time convincing them that 
this is what we needed to do, the workshop helped them realize they were ready to do it. 
And that was the turning point. They finally decided they could really do team science 
projects all year long and then go to the science fair. And through the consortium’s help, 
and through staff and the Tribal Coalition, all eight schools came together to do a science 
fair for the community, and we ended up with over 900 kids, and just couldn't believe the 
success in it. And the group has survived, and this February we'll have a second science 
fair, and we're going to make it a two-day event instead of a one-day, so that the schools 
can tour and all that. 

— Vignette #61 

An elementary school teacher from the Northeast and Islands Region also spoke of the 
strengthened relationships between teachers in their region: 

As an intensive site, we got to network with other teachers from our state and from all of 
the consortium schools at one state meeting and two regional institutes. It was really 
great to get together with the state team, and share ideas ... it was wonderful just to share 
success stories, or, you know, what didn't work. Especially when we were just starting 
out trying to write a curriculum and we were able to get together with people who had 
done it. And just for them to say, “try this, try this, but don't bother with that.” So it was 
really good to have a chance to talk with colleagues in that kind of forum. I mean, just to 
spend four days with people who are really excited about improving their schools was 
uplifting for our whole team. 

— Vignette #62 

The following comments from a state supervisor in the Mid-Atlantic Region also illustrate 
strengthened relationships that enhanced working relationships and professional alliances: 

Through the state team, we can deliver professional development regionally (across the 
state). An enormous amount of planning is involved in the regional delivery of 
professional development. One regional collaborator talked with another region to build 
his model for alliance. The coordination for delivering professional development and 
cooperation among districts has increased. 

— Vignette #63 

2. Reform Synergized 

In addition to client comments on strengthened relationships, many also spoke of reform or 
improvement efforts they have seen in their region as a result of the collaboration and networking 
activities of their regional consortium. Regional consortia contribute to systemic reform by 
convening key stakeholders to collaborate and plan efforts. The following highlights from client 
interviews illustrate the Network’s impact on reform efforts. 

A mathematics coordinator for a state department of education in the Appalachia Region spoke of 
reform efforts in this region: 

I am on the consortium’s regional board and a state steering committee. We have met 
throughout the years to focus on mathematics and science reform issues. These meetings 




35 



43 



serve as a key way for us to network. This type of networking strengthened our ability to 
better enhance the reform effort in our state and the region. By bringing the key players in 
mathematics and science reform together (across the region and within each state), we each 
shared what we were focusing on and we learned about the big picture. Anytime you have a 
guided opportunity to reflect and share, you come away more aware of the big picture. My 
participation in these activities reinforces or gives me direction that we are heading down the 
right path in terms of mathematics and science reform. We learn that our services and 
products are going in the right direction. In addition to their efforts to convene and connect 
stakeholders, they also link us with a sub-set of a national program. We have our state and 
regional perspectives, but the consortium brings us out of “us” and allows us an opportunity to 
look at the national picture and how we fit in. Without the consortium, we would not have 
been able to meet and make the connections across our organizations. 

— Vignette #64 

And, a state science supervisor from the Northeast and Islands Region also spoke of the impact 

the consortium has had on systemic reform efforts in this region: 

The consortium has been a valuable resource in many ways, especially having access to 
their expertise, but for me, it’s mainly meeting individuals through consortium activities 
that has been most useful. In the last couple of years, the consortium director set up a 
series of meetings with the Eisenhower Coordinators from all of the states in our region. 

To me, it was a support opportunity when the Eisenhower Coordinators would meet, 
because we were struggling at a time when we were doing consolidated applications, and 
none of us were sure what that meant, and there was fear that the science and math 
professional development piece would be lost in that whole attempt. And so those 
meetings . . . were very, very crucial at that time. The knowledge that I got from those 
meetings and other regional conferences I've been involved in, all helps me leverage 
change, and we've had dramatic changes in things such as developing standards for 
schools, developing new standards for pre-service teachers in the state, leveraging the 
necessity to have state testing of teachers in science content areas. And you know, a lot 
of these Averagings came about through meetings and collaborations with people who 
participate in the network that the regional consortium helps support. The consortium 
provides me with a resource that I can use to help me gain more knowledge or resources, 
and for me, that makes me more effective. 

— Vignette #65 

Many collaborators also attended professional development sessions offered by the consortia. 

One higher education faculty from the Southwest Region described how these sessions enabled 

participants to network across multiple states: 

I do pre-services for teachers — I teach teachers-to-be. . .Because of attending the Fall Forums 
(professional development sessions), I have made connections with people across the five 
states — with people in various capacities — they were all very, very different even though they 
were all math and science. There were also people from different universities and school 
districts — all at different places in their reform movements. We've been able to take our 
individual ideas and develop something even bigger and better from a conglomeration of the 
ideas. It introduces people, and agencies, and other people to each other so that they will 
know how to get things done the next time they come up. And they keep us all current on 
what is new and important. 

— Vignette #66 
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A curriculum director from the North Central Region provided an example of how the Network’s 
efforts have supported regional reform: 

We have been designated as an intensive site for the past four years. The consortium has 
collaborated with the district and provided technical assistance in a variety of ways, both in 
mathematics and science. They have assisted us in delivering the staff development and were 
instrumental in doing an analysis of our curriculum and making sure that our curriculum was 
in line with the state standards. This collaborative relationship has enabled the curriculum 
department in our district to coordinate the mathematics and science curriculum so that we are 
not operating in a silo with each of those programs. Our partnership with the consortium has 
brought credibility to our curriculum. It has allowed our board members to ask some really 
hard questions about curriculum. The board has approved both the math and science 
curriculum frameworks. That sends me a message that we feel we are on the right path and 
the board appreciate the partnership we have with the consortium. 

— Vignette #67 

Mathematics and science initiatives in the Mid-Atlantic Region, too, have been impacted as a 
result of consortium support. A curriculum coordinator from this region characterized the impact 
as follows: 

The consortium has provided access to resources and allowed us to work with schools because 
we can say we are part of the large consortium. Through networking on the team, my staff 
have become known to the state department of education and have gained access to resources 
for mathematics and science initiatives. In sum, collaboration on the state team increased 
communication across the state, created alliances and partnerships with informal science 
institutes, facilitated work with the state department curriculum supervisor, all of which would 
not have happened as easily without the consortium. 

Participation on the team has had immense impact on the work of this organization. We 
became an ENC Access Center and, most recently, purchased two STAR LABs, a 
portable planetarium, and now are running our tails off providing professional 
development. Teachers are using it, and it’s made its way into schools’ curriculum. 

We’ve put programs into schools that wouldn’t be there without the professional 
development in the collaboratives that were developed and supported through the state 
team. 

— Vignette #68 

3. Access to Resources Increased 

The Network’s collaboration and networking activities have also brought about increased access 
to resources and leveraging of resources, as was previously mentioned by a curriculum 
coordinator from the Mid-Atlantic Region. Evidence of this can also be found in the following 
excerpts from client interview data. 

A principal investigator for a systemic reform initiative from the Far West Region commented on 
the leveraging of resources brought about by their collaboration with the regional consortium: 

When we first started this NSF-funded systemic initiative in the developmental stage, we 
organized an advisory board, of which the consortium was one of the first members. 

Because the initiative represents 28 tribes in four states, and we’re very rural, this led to a 
very complex set of needs that we had difficulty addressing very well until the 
consortium came in to help us with the transition to a partnership council. One of the 
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reasons why we evolved into a partnership council was to try to coordinate services. By 
facilitating meetings and bringing together the professional development providers, the 
consortium was instrumental in helping us formulate and keep the partnership council 
alive. This helped bring the resources to bear on the needs of initiatives. I mean there 
were major changed in our ability to do things as a result of that. For example, we got 
many resources from our other partners having to do with professional development on 
curricular alignments, professional development on leadership training in particular. 

By helping us coordinate our partners, we got much more direct benefit to our coalition 
and schools because we were able to cull from the partnership better focused resources 
that we hadn't identified before. I think that having the consortium as a major partner 
increased our efficiency as an organization, and thereby we were able to impact more 
quickly reform at the local level. So I think that this early intervention really helped move 
the reform process along. 

— Vignette #69 

A principal for a middle school from the Southeast Region described how their collaboration with 
the consortium leveraged resources to assist the building of a multi-million dollar science center: 

When we were building our environmental center, the consortium was able to connect us 
with people that had expertise in different areas, like environmental concerns and those 
kinds of things, so that we were able to get their assistance in planning and developing 
and building that center. The environmental education center which is actually about a $6 
million facility that we would not have been able to build without the consortium's 
assistance and other partners. But I think just having the consortium as that base to build 
on, and to say that the consortium was behind this project, and they were helping us to 
develop it, brought a lot of the other partners in and helped us to acquire the resources 
that we needed to build the facility and to carry out the mission that we had started with. 
— Vignette #70 

Increased access to resources was also highlighted by a public school principal from the 
Southeast Region: 

We would all be out there struggling on our own without some guidance and leadership. 
The consortium brings us the leadership and brings us together so that we can have the 
coordination. We’re not all duplicating resources in each of our separate regions, but 
we’re sharing and learning from one another. I learned about math and science 
approaches and directions they were taking and initiatives, and brought those ideas to my 
faculty. Through our sharing process and our networking at board meetings, I was able to 
leam about resources that were being used in other districts in other states, and then call 
upon those resources myself. For instance, I was receiving the brochure on the 
clearinghouse (ENC) materials, but I wasn’t familiar enough to know how you can use 
and order these materials. And through networking with others, I learned about resources 
that were very beneficial at the school level. 

— Vignette #71 

A state science supervisor from the Northeast and Islands Region also noted how access to 
professional development and print resources from the consortium enabled participating districts 
to coordinate their efforts: 

In the eight years that I've collaborated with the consortium, they have assisted us in 
ways that have had a tremendous impact on the services we provide to the school districts 
in our state. Because of the consortium, we have been able to provide our state with print 
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resources and comprehensive professional development that we wouldn’t have been able 
to otherwise. The one really significant effort that was most helpful to me was a series of 
four workshops that we were putting on, that the consortium director helped us plan and 
organize, and provided direct services and resources. That had a tremendous impact 
because we had over half the districts in the state participate in those four workshops. It 
was phenomenal. It gave them an opportunity to network, but also to coordinate efforts 
that schools in districts were doing in isolation from others in the same district. And the 
consortium provided all the districts with copies of the Professional Development book, 
and the Equity Manual that our State team created the year before, as well as copies of 
ENC Equity CD, and all of the rich resources that come out of the Eisenhower 
Clearinghouse. 

— Vignette #72 

Thanks to resources accessed through the Network, educators are able to do their jobs more 
effectively, as evidenced by the following statement from a state-based professional development 
provider in the Northeast and Islands Region: 

Last spring, I attended a two-day workshop on Case Studies so that I could share that 
approach to ongoing professional development with the teacher leaders I mentor. It was 
wonderful to have an opportunity to participate with other professional developers, or 
people who work on a larger scale than just the school or district scale, from other parts 
of New England. In that way, the consortium holds a unique role, in that there isn’t a lot 
out there for support and professional development for the professional developers. You 
know, it’s an isolated bunch. And it’s those networking opportunities that make it 
possible for all of us to do our jobs better. 

— Vignette #73 

As confirmed by these interviews and survey data collected in previous years, the Eisenhower 
Network has had a significant impact on collaboration and networking among its clients. In 
addition to strengthening relationships and supporting reform efforts, the Network’s collaboration 
activities have resulted in greater access to resources and increased service coordination. These 
collaborations are valued by stakeholders: over 80 percent of interviewed clients feel that the 
collaboration has met their expectations and/or provided them with benefits not otherwise 
afforded. 
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IV. Conclusions 



The Eisenhower Network evaluation was undertaken to describe the activities and impacts of the 
regional consortia and national clearinghouse program, as well as to highlight areas for 
improvement and change. Overall, the evaluation findings indicate that the Network has taken 
great strides toward enabling effective reform of mathematics and science education. The 
Network has been delivering high-quality services to educators across the country on a large 
scale, and Network products and services have been rated highly by clients. However, the 
evaluation has also been helpful in identifying possible areas for improving performance, such as 
increasing the proportion of intensive services. This final section of the report focuses on overall 
conclusions from the five-year Network experience. 

The most global learning from the Network experience is that moderately funded entities with 
expansive mandates can establish themselves and survive as productive members of the 
mathematics and science community. This finding may seem obvious in 2001, but it was not so 
predictable in 1992 when the Network was first funded. Just gaining “a place at the table” within 
this already well-defined community is an accomplishment that is attested to by the Network 
advisory groups, clients, partners, and evaluation results. 

In addition to confirming the quality and effectiveness of particular consortia services, the 
evaluation, particularly the client interviews, revealed an overarching theme: the Eisenhower 
Network serves needs that no other entity has been able to address. Its services are perceived as 
uniquely valuable. Specifically, the interviews identify the combination of national perspective 
with regional knowledge and positioning as the key to Network effectiveness in furthering 
mathematics and science education reform. 

The interviews identified four services the Eisenhower Network provides which clients could not 
find elsewhere: 

• offering a national perspective and access to national resources and expertise; 

• providing reliable, targeted, and high-quality professional development on broad reform 
topics , in response to regional needs, building capacity at all levels; 

• connecting state-level policy makers, regional service providers, and district and school- 
based educators, leading to lasting working relationships that cross district, state , and 
regional boundaries , thus improving their capacity to do their jobs; 

• serving high-need schools that no one else serves , those too small or poor to access expertise, 
or so politically embroiled that they require an unbiased outsider. 

1. National Perspective and Support for Mathematics and Science Education Provided 

The national perspective of the Network is highly valued. As a state department of education 
mathematics coordinator from the Appalachia Region said: 

The consortium links us with a sub-set of a national program. We have our state and 
regional perspectives, but the consortium brings us out of “us” and allows us an 
opportunity to look at the national picture and how we fit in. 

— Vignette # 74 

A state science coordinator from the Far West Region also highlighted the importance of the 
Network’s regional and national perspective, and went on to illustrate that, positioned as it is 
beyond local and state politics, with extensive knowledge of regional resources, the Eisenhower 
Network is able to serve as an effective advocate for mathematics and science reform at the state 
and regional levels. 
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When the consortium director was helping us to revise our state's core curriculum, he brought 
a significant amount of expertise and resources from Western states, as well as a national 
perspective, and gave us kind-of an opportunity to have someone stand back and look at what 
work we were doing. The consortium supplies us with direction in looking at science reform 
with a wider lens than what we typically do here. When we work in our own little circle, it 
becomes very provincial. Having individuals who are looking at curriculum and resources on a 
national level is extremely valuable. It provides us with the perspective of a national resource, 
to make sure that we have things that are, in fact, on that scale. When we need information 
related to what's transpiring in other places, they're always good with that. 

Two years ago, the state legislature made a decision to remove science assessment from 
the overall assessment picture. Math and reading became the only things that were 
assessed, and it created quite a stir. But the regional consortium provided our districts 
with the resources to look at other states, and without any input beyond that, our district- 
level people went to their legislators, to their professional organizations, and reasoned 
with legislators so that last year that decision was reversed, and now the end-of-level 
criterion-referenced test for science has been restored as a result of that. It was quite nice 
to have someone provide information about what's going on in other states, to inform our 
district people of what sort of voice should be used to reason with our legislature. 

Overall, in general I think the consortium has addressed needs that other organizations 
wouldn't have addressed without consuming much of our own resources to accomplish, 
and I don't think other organizations would have been consistently there the way the 
consortium has been. 

— Vignette #75 



In their interviews, many clients also conveyed satisfaction with the advocacy role the Network 
fills for them at the national level. The Eisenhower Network, they said, provides their district or 
state a voice at the national level, opening doors to national expertise, which they don't feel they 
have otherwise, and connecting them to a national network of colleagues. 

A state department of education mathematics consultant from the Mid-continent Region 
explained: 

As the consortium staff talk and collaborate with us, I know that they summarize and 
share the concerns that we have as state supervisors with representatives from the U.S. 
ED (U.S. Department of Education) and ENC (Eisenhower National Clearinghouse). I 
think it fills the niche of helping direct some funds and resources to individual state 
needs. They can bring some national perspective, but be flexible, helping us leverage the 
national initiatives and tailor them to meet our individual state needs 
— Vignette #76 

2. Capacity Built through Professional Development 

Although the national perspective and advocacy were identified as important, client interviews 
revealed that the Network is more than a reliable pipeline to the latest information and expertise. 
The Network watches closely for needs that present themselves across multiple states, and 
strategically provides regional professional development on critical education reform topics in 
which states do not have expertise. In this way, the Network builds state and regional capacity, 
and saves states and other providers from duplicating efforts. 
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A district curriculum supervisor from the Mid-Atlantic Region, who is a member of a state team, saw 
her participation as providing unique access to important national topics and experts for herself and 
teachers from her district: 

The consortium holds its own professional development, such as the TIMSS conferences, 
which are really valuable, bringing professionals from the whole region together. I 
would never have known about the TIMSS conferences, and teachers would never have 
gone. How else would you get a multi-state TIMSS conference going? It would never 
happen. . . It's hard for small states to get Bill Schmidt and speakers from AAAS and 
NSF; we’re now more likely to have access to those kind of people. Administrators are 
better informed. As a science supervisor at the district level, you knew something was 
coming. I learned about the best, highly recommended resources. The TIMSS 
conferences constantly stimulated science educators to think about what’s important. The 
consortium should continue, as it does fill that role. 

— Vignette #77 

A regional professional development provider from the Northeast and Islands Region elaborated 
on the importance of having access to the consortia expertise on such as topics as strategies for 
using data: 

Because of their regional role, and being postured with the U.S. Department of 
Education, the consortia have a solid grasp on some of the issues that we haven't had time 
to wrestle with. I think there's a need for broad education reform issues to be handled by 
the consortium. Specifically, to have the consortium staff work with our districts on data 
was really helpful. I think they found the meat in that to make it practical for districts. 

There are very few people that have anything of quality to offer around data. There's a 
screaming cry for it; there's a screaming cry for everybody to use it, but I will tell you 
that so many of us - and I don't just mean the classroom, I don't just mean teachers, I 
don't just mean districts, I mean even our organization - we have data everywhere. We 
have more data than we could go through in ten million years. So, to have an opportunity 
to have someone help you focus your energies on what is important - not because 
somebody else said it was important, but what is truly important, for you as an 
organization, for you as a district, for you as a classroom teacher ... no one else is filling 
that niche. 

— Vignette #78 

3. Capacity Built by Forging Connections across Boundaries 

The theme of appreciation for the Network’s external position emerged from the client interviews in 
multiple forms. As illustrated above, the Network has expertise and political impartiality that helps it 
to serve as an effective advocate. The regional position of the Network also makes it uniquely 
effective at crossing organizational and geographical boundaries. The Network connects local and 
state educators to resources and expertise within their own state and across state lines, and provides a 
broad perspective and framework that builds capacity by helping people work together to solve 
common problems. 

The director of a NSF-funded technical assistance provider in the Appalachia Region explained 
how the Network is helpful at the state level: 

The niche filled by the consortium is their ability to convene and connect stakeholders to 
work collaboratively on systemic reform. The consortium is an organization outside of 
the state that can come into the state and assist with systemic reform. This is a valuable 
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resource because they are able to bring a different perspective and to provide a broader 
view in terms of the national context. The consortium has also been able to connect key 
organizations within the state in a way that no state-specific organization can do. The 
state department can’t bring all the key players together. They have tried. Only the 
consortium has been able to do this successfully. 

— Vignette #79 

A higher education mathematics faculty member and coordinator of a mathematics and science 
center for professional development in the Mid-Atlantic Region was able to enhance the quality 
of his professional development, also as a result of the Network’s support at the state level: 

The consortium, through the state team, provides necessary frameworks to get people 
across the state to talk about common problems, share expertise, ideas, help all of us to 
do a better job of what we do. It lets us do our jobs, but helps us do them better. What 
the consortium does that no other organization can do is to bring all of the different kinds 
of organizations and programs to the table and to be unbiased. Also, I know there have 
been real benefits from the matching money provided by the consortium. Without the 
matching money, these professional development programs would not have happened. 
Each year, we are addressing more needs. It is helping us do our job. 

— Vignette #80 

Brokering collaborations within a state clearly builds capacity, but equally important are the 
relationships the Network facilitates across state borders. As expressed in many client interviews, 
sometimes these long-distance collaborations provide support not available to educators within 
their own states. This was illustrated by the state science supervisor from the Northeast and 
Islands Region quoted above on page 36 who valued the support he received from meeting with 
the other Eisenhower Coordinators in his region at a time when they were all struggling with a 
new challenge, consolidated applications. In his words, the knowledge that he's gained from 
regional conferences and meetings like those have helped him "leverage change." 

Even those with a ready network of colleagues at the state level appreciate the richness that only a 
regional forum can provide. The Southwest Region higher education faculty quoted above on 
page 36 expressed a common theme among the client interviews when he said that one of the 
things he finds wonderful about the consortium is that it involves multiple states. As he says, 
there is value in making connections with people representing a wide range of capacities, "all at 
different places in their reform movements." Many interviewees cited such connections across 
geographical boundaries as adding value to their work in this way. 

4. High-Needs Schools Served 

Several client interviews provided a district and school-based perspective on the importance of 
having a regional, external source of support. As they articulate, it is critical to have a provider 
like the Network who can serve high-need schools without concern for profit or politics. The 
following statement, shared by a district science supervisor from the Appalachia Region 
highlights the niche consortia fill by serving resource-poor schools: 

The training that has been provided to the teachers and schools who might not otherwise 
have been able to afford it has just been unbelievable. A lot of these teachers who work in 
poorer school districts might not ever get any professional development if it weren’t for 
the consortium and the training they provide. 

— Vignette #81 
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In some cases, the Network’s expertise is what qualifies it to work with high-need sites that 
others do not. The principal investigator of a rural systemic initiative in the Far West Region 
shared this example: 

The consortium helped to develop and implement the Tribal Innovation Program, a two- 
year teacher training program that focused on the alignment of curriculum and 
instruction, standards, and assessment. This program worked primarily with BLA schools 
and on the reservations. I think it would have been difficult for another organization to 
work with the Native American districts that we work with. I think a lot of the other 
organizations either didn't feel competent to work with them, or were restricted for other 
reasons from working with BLA or native schools. The students who benefited from the 
teacher training, the alignment and so on, were almost 100 percent Native American. We 
also had a Hispanic Issues conference, and we also got some support from the consortium 
on not only preparing that, in fact, staff were very instrumental in acting as a facilitator 
for organizing that institute, but they also offered support at the institute itself. And from 
that, we've come up with a series of “Issues” documents that we're putting into a 
“Proceedings” document that I think would be a very fine guide for science/math reform 
for Hispanic populations. That guide is being developed now. 

— Vignette #82 

In addition to direct service by the Network, a district supervisor from the Mid-Atlantic Region 
illustrated the way state teams and regional collaboration also serve to focus attention on high- 
need groups: 

As a member of the state team and regional board, I made sure teachers of urban and 
minority students had a voice. The state meeting helped the team establish a vision for 
what they wanted to accomplish and developed a plan for how to accomplish goals. I 
represented urban teachers and the urban child. The team had a pivotal role. We became 
a political sounding board, picked up what any one organization couldn’t do, like a 
survey on equity issues. Before the team, I hadn’t realized the state was so rural. I have 
a better awareness of the state of education in the state and have established many 
collegial relationships. I have a support network that I never would have gotten 
otherwise. 

— Vignette #83 

A Final Word 

The national expertise and regional focus of the Eisenhower Network enables it to promote 
mathematics and science reform in many ways identified by their clients. The network offers a 
national perspective and serves as a reform advocate for state and district educators and policy 
makers. It builds capacity at the regional, state, district, and school-building level by providing 
targeted, high-quality professional development and technical assistance, by connecting educators 
across geographical and organizational boundaries, and by serving high-need schools whose 
needs have not been met effectively by other entities. In these ways, the client interviews indicate 
that the Eisenhower Network provides unique and valued services to educators at all levels. 

Because of its substantial investment in evaluation, the Network can present a clear picture of 
what has been accomplished in the 1995-2000 grant period. In the coming months, the Network 
directors will use the findings of this report to shape future study and activities, as a new five-year 
period of service to the mathematics and science education community is launched. 
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Appendix A 
Evaluation Methods 



Evaluation staff representing each of the ten consortia and ENC have worked collaboratively to develop 
the Consortia and Clearinghouse Descriptive Data System (CCDDS), the participant survey, and the 
client interview to gather data regarding basic evaluation questions on the operations and outcomes of the 
Network effort. The guiding evaluation questions were as follows: 

• What is the portfolio (i.e., variety, scope) of services offered by the Network? 

• Do Network contacts with clients through print and/or electronic media increase annually? 

• Does the Network reach substantial numbers of practitioners and policy makers (stakeholders in the 
educational community) with a variety of services? 

• Is the Network targeting LEA clients who are working with at-risk students (e.g., students eligible for 
free or reduced price lunch)? 

• What are the levels and kinds of collaborator involvement in Network activities? 

• Are Network activities aligned with National or State content and performance standards and focused 
on assisting in the implementation of these standards and practices related to their attainment? 

• Do clients perceive that the Network provides quality services? 

• How are Network products and services used and what are their perceived values/utilities? 

• To what extent do the Network’s products and services contribute to clients’ work? 

• What percentage of participants in trainer-of-trainers activities report that they go on to provide 
professional development or technical assistance based on the technical assistance they receive from 
the Network? 

• What are the impacts of products/services on: 

- Professional practice of service recipients; 

- Student performance at sites in general, as indicated by teacher reports; 

- Student performance at intensive sites in particular, as indicated by student assessment 
scores; 

- At-risk, underserved students; and 

- Collaboration/networking/partnership activities? 

• What mechanisms are in place to assure the quality of consortium products and services? 

• What are the apparent strengths of the Network? 

• What challenges does the Network face in accomplishing its goals? 

Descriptive information about the consortia activities has been collected since October 1995 through 
ongoing use of the CCDDS. ENC began participating in the CCDDS in January 1998. Evaluative data 
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regarding the perceived quality and impact of these activities were collected in 1997, 1998, and 1999 
using a participant survey, and in 2000 using the client interview protocol. This section summarizes the 
evaluation methods used. 

A. Consortia and Clearinghouse Descriptive Data System (CCDDS) 

The CCDDS captures extensive descriptive information documenting client-service activities along a 
number of dimensions including: major service category, content focus, level of collaboration, method of 
service delivery, activity duration, number of participants, and client/collaborator organizational 
affiliation. A sample of a CCDDS record form is shown in Appendix D. Evaluation staff at each 
consortium and ENC gather the information, enter it into the CCDDS database using FileMaker Pro 
software, and provide electronic files for cross-consortia aggregation. Data were checked for accuracy 
and completeness, aggregated, analyzed, and summarized for this report. 

B. Client Survey and Interview 

To present a more comprehensive picture of the Eisenhower Network program, a companion evaluation 
strategy was developed to focus on an assessment of quality and impact of the consortia and ENC 
activities. A pilot survey instrument was designed in 1997 to include a common set of questions for 
participants regarding training and technical assistance, dissemination, and network building, with some 
flexibility to tailor the instrument to the needs of each consortium and ENC. For example, organizations 
were able to include questions regarding their own products and collaborative groups. Based on the 
results of the pilot test, the survey instrument was revised in 1998 and again for 1999. 

To obtain additional client data for the Network’s five-year evaluation report, the Evaluation Committee 
decided to design and conduct client interviews in 2000 instead of administering the annual mail survey. 

It was felt that this change in methodology would enhance the Network’s understanding of client needs 
and, consequently, its ability to improve services. The interview was designed to address selected OERI 
performance indicators, as was the survey. However, in contrast to the survey, the interview was expected 
to yield richer, more in-depth qualitative information that is particularly pertinent to client 
outcomes/impact and lessons learned during the grant period. Each of the 1 1 organizations in the Network 
conducted an interview study. The interview methodology is described below including: (a) interview 
protocol development and pretesting; (b) sampling guidelines and procedures; (c) interview 
administration; (d) data entry and analysis; (e) external audit; and (f) report development and quality 
assurance (refer to Appendix B for samples of interview materials). 

Interview Protocol Development and Pretest. The Evaluation Committee formed a Client Interview 
Subcommittee to design the interview and guide its implementation. The Subcommittee developed the 
interview protocol and cover letter, sampling guidelines, procedures for data entry, analysis, and 
reporting, and other aspects of the interview methodology. The full committee reviewed draft versions of 
materials at key points in the development process. The contact letter and protocol were pretested with a 
small sample of clients and consortia/clearinghouse staff. At the end of the each pretest interview, the 
respondent was asked to provide feedback on the clarity of the interview questions, the meaning of key 
words and interpretation of selected questions, and how the interview could be improved. The protocol 
was then revised based on this feedback. The final protocol consisted of four sections including items 
pertaining to background information, training and technical assistance, collaboration and networking, 
and overall ratings of the consortium and clearinghouse. 
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Sampling Guidelines and Procedures. A goal was established for each of the 1 1 participating 
organizations to complete 16 interviews with a sample of clients involved in Year 5 technical assistance 
activities and/or collaboration/networking activities. Each sample was to include at least eight clients who 
are, as described in Indicator 1.3, “teachers, administrators, and providers of professional development 
who participate in the Consortia’s continuing technical assistance” and at least eight clients who are, as 
described in Indicator 1.9: “participants in consortia partnerships, collaborations, teams, and networks.” 
However, because ENC does not provide training/technical assistance, it was later determined that ENC’s 
sample would include only eight clients from the latter category. 

Additionally, staff were asked to keep in mind six sampling considerations which recommended the 
selection of clients with greater intensity, longevity, and breadth of involvement with the consortium or 
ENC; and the selection of a broad range of professional roles, to include clients who work with at-risk 
populations, and to include professionals from both the math and science education communities. These 
considerations were included to help ensure that respondents had a greater degree of experience with the 
Network and could therefore give richer, better informed responses and also address the organization’s 
strengths as well as areas for improvement. The sampling considerations were also intended to ensure that 
the diversity of the Network’s client base was adequately represented and that the sample would reflect 
the work carried out with these different populations. Finally, the sampling guidelines recommended the 
use of “purposive random sampling” to address the above criteria and considerations within the 
constraints presented by the small sample size (n=16). Within this general framework, the guidelines 
provided some flexibility so that each evaluator could tailor the sampling procedures to each 
organization’s circumstances as needed. In general, the samples were selected in one of two ways: (a) by 
identifying intensive activities in the CCDDS, identifying participants in these activities based on the 
sampling guidelines and considerations, and then randomly selecting client names from the pool; or (b) 
asking program staff to nominate clients who would meet the sampling guidelines and considerations and 
then randomly or purposively selecting client names from the nominations. Several organizations also 
stratified the sample by Criteria 1 and 2 and/or by state, and some identified extra names for use as 
replacements in case anyone in the initial sample could not be reached or declined to participate. 

Interview Administration. Potential respondents were informed of the interview by an advance letter 
which was generally sent by email or fax. Most of the respondents were later contacted by telephone to 
schedule the interview. Respondents were assured of confidentiality and that responses would be 
reported in aggregate form as specified in the advance letter and/or interview protocol. If quotations were 
desired for the evaluation report, identifying information would be removed or else clients would be 
contacted for permission to quote. Interviews were generally conducted by the evaluation staff, although 
two organizations hired other firms to conduct interviews for them. Interviews were generally conducted 
by telephone and tape recorded and/or notes were taken by the interviewer. One organization conducted 
some interviews by phone and some in person, and one organization conducted the study during a Math 
Science Leadership Team meeting. In the latter case, participants received copies of a reformatted 
protocol, responded to the questions in writing, and followed up with the interviewer to clarify the written 
responses. 

Across the consortia and ENC, 163 interviews were completed. Eight of the ten consortia completed 16 
interviews each, one completed 17, and one completed ten interviews; ENC staff completed 8 interviews. 
The first interview was completed on September 20 th and the last interview was completed on November 
25 th , 2000. Nearly all of the interviews were completed by the end of October, but one consortium 
completed all of its interviews during the month of November. 
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The sample consisted of 163 clients from all 50 U.S. states plus 7 U.S. territories/districts/possessions. 
Most of the clients interviewed (91%, n=149) first became involved with their regional consortium or the 
ENC in 1 998 or earlier, and approximately half of the overall sample (48%, n=78) had been involved 
since either 1995, 1996, or 1997. With regards to discipline, nearly half (48%, n=78) of the clients 
interviewed consider their work to be related to both math and science education; 22% (35) consider their 
work to be related specifically to math education; 29% (47) to science education; and 2% (3) were unsure. 

The following table depicts the distribution of interviewed clients among employer affiliations. Overall, 
the most common employer affiliation is Public School District (40%, n=65). When this category is 
combined with the Private School District category, the number of clients affiliated with a school district 
increases to 44% (71) of the overall sample. The next most common employer affiliation is State Agency 
(20%, n=32); each of the 1 1 Network organizations interviewed at least one state-agency client. The third 
most frequent affiliation is Institution of Higher Education (17%, n=28). The remaining 19% consists of a 
mixture of affiliations such as National Science Foundation (SSI/USI/RSI/LSC), Professional 
Association, Intermediate/County Education Agency, Business/Industry, Regional Service Provider, and 
Federal Agency. 



Table 1. Affiliations of Interviewed Clients 



Employer or Affiliation 


Number of Clients 


Percentage 


Public School District 


65 


40.1% 


Private School District 


6 


3.7% 


Institution of Higher Education 


28 


17.3% 


Professional Association 


5 


3.1% 


Business/Indus try 


3 


1.9% 


Federal Agency 


2 


1.2% 


State Agency 


32 


19.8% 


Intermediate/County Education Agency 


4 


2.5% 


National Science Foundation (SSI/USI/RSI/LSC) 


6 


3.7% 


Regional Service Provider 


3 


1.9% 


Other 


8 


4.9% 


Total Valid Responses 


162 


100% 



The 71 school district-affiliated clients were asked to identify their primary professional role. Among this 
subset of interviewed clients, the most common school district-affiliated role is Curriculum/Content 
Specialist (31%, n=22), followed by Administrator (21%, n=15), and Elementary Teacher (18%, n=13). 
Figure 2 illustrates the extent to which the school district-affiliated clients in the sample are distributed 
among professional roles. 



Table 2. Professional Roles of School District Clients Interviewed (n=71) 



Professional Role 


Number of Clients 


Percentage 


Elementary Teacher 


13 


18.3% 


Middle/Junior High School Teacher 


6 


8.5% 


High School Teacher 


7 


9.9% 


Other Teacher 


2 


2.8% 


Curriculum/Content Specialist 


22 


31.0% 


Administrator 


15 


21.1% 


Other 


6 


8.5% 


Total Valid Responses 


71 


100% 
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Approximately half of the school district-affiliated clients that were interviewed serve majority at-risk 
student populations, as shown by the following findings. 52% (34) of the 66 valid responses from school 
district clients indicated that at least 50% of students in their school or district are eligible for free or 
reduced-price lunch. In addition, 52% (36) of the 69 valid responses from school district clients indicated 
that a majority of their school or district’s students are Title I, Native American, Pacific Islander/Native 
Hawaiian, or LEP. (The number of valid responses was calculated by taking the total number of school 
district-affiliated clients, 71 , then omitting those who were not sure or did not provide a response.) 

Data Entry and Analysis. Interview data were generally transcribed from audio tapes and/or generated 
from the interviewer’s notes. The data were entered by each organization into a common data 
spreadsheet in Excel software. Some staff later imported their data into a common Filemaker Pro 
database template to facilitate analysis. Evaluation staff reviewed and edited their own data prior to 
analysis. Each evaluator coded the closed-ended data, referring to specified data coding procedures. The 
closed-ended data were then aggregated into an Excel spreadsheet so that Network-wide trends could be 
identified and analyzed. Closed-ended data analysis procedures were primarily descriptive of categorical 
frequencies for the quantitative data. 

Initial plans called for each evaluator to also code the open-ended data according to common data coding 
procedures. As originally conceived, the subcommittee had intended to compile a set of open-ended data 
codes after the committee had held some preliminary discussions via an online discussion forum to 
identify the common themes and rare responses. To allow more time for evaluators to become familiar 
with their data, the plan was changed to have these conversations take place at an in-person meeting, 
where the committee worked in teams to distill a set of common response themes-with codes--for each 
section of the interview. Following the meeting, one person from each team pilot-tested the codes on 
their own interview data, then the team members re-conferred to make any necessary revisions. One 
example of a revision was to add sub-thematic codes to signify the quality of impact; another was to add 
examples and directions. The evaluators then used the complete, revised coding protocol to code their 
open-ended interview data. 

External Audit. A hard copy of each organization’s cleaned, coded open-ended data was submitted to an 
external evaluator who was asked to review and audit the codes for consistency and reliability (refer to 
Appendix C). The audit revealed that the coders had interpreted and applied the coding scheme in a 
variety of ways. Based on the external evaluator’s recommendations, the committee decided against 
using the codes quantitatively, but instead to use them as markers to help locate relevant examples for use 
in the final report. These examples were used to generate client-specific vignettes that illustrate the 
indicators and the Network’s unique role in improving mathematics and science education while also 
representing the different client populations. Using a set of specified procedures as a guide, each 
organization developed a set of vignettes from its interview data, and constructed each vignette using a 
combination of direct quotes and/or paraphrased text from each client interview. Each vignette was 
intended to represent the salient aspects of a client’s interview. Wherever possible, each vignette 
described the client’s context as well as the outcome, population affected, and process by which that 
outcome was brought about (i.e., the nature of activities in which the client was involved). 

Report Development and Quality Assurance. Members of the Evaluation Committee formed subgroups to 
further review the vignettes, and to incorporate a subset of them into an initial draft of this report. Draft 
sections were then checked for accuracy, compiled into one document, and edited by several members of 
the Evaluation Committee. A working draft report was produced and circulated to the evaluators and 
directors for their review and comment. The review process included the use of a quality assurance 
review form and followed a procedure that had been developed for prior Network evaluation reports. 
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C. Value and Limitations of the Data and Lessons Learned 

It is important to note that although efforts have been made to apply the database coding system and client 
interview coding scheme consistently across all consortia and the clearinghouse, there have been some , 
differences in interpretations of the coding scheme over time and across the organizations. The consortia 
evaluators have taken steps to reduce possible discrepancies by identifying areas of potential 
misunderstanding, meeting regularly to discuss the coding scheme and its application, and working with 
consortia and clearinghouse staff to increase uniformity of procedures. 

The two data collection methods and related instrumentation were developed for different purposes and 
varied greatly in scope. Results of using both methods are included in this report, but they may not have 
the same levels of validity and reliability as means of representing the Network activities and impact. 

Data have been reported using two different units of analysis: (1) activities, and (2) participants within 
activities. Both units must be considered to obtain a comprehensive and accurate understanding of the 
consortia and clearinghouse work. 

Data collection for the CCDDS uses program activities as the unit of measurement. Activities are coded 
according to service category (“what consortia do,” e.g., provide technical assistance) and focus area 
(“what the activity is about,” e.g., standards-based instruction), among other characteristics. Each activity 
may be coded with up to three prioritized service categories and focus areas. For this report, the first- 
priority service categories and focus areas are represented in all analyses. In some cases the second and 
third priorities are also included to provide fuller descriptions. Including only the first priority may 
under-represent the focus areas; including all three may be hard to interpret as it introduces focus-area 
overlap among activities. In addition, the database does not allow for the identification of unique 
participants; multiple client contacts may involve the same individuals. 

Data collection for the interview uses participants as the unit of measure. The established goal for this 
year’s study was to obtain 16 completed interviews from each of the 10 consortia and eight from ENC 
(i.e., 168 interviews). The completed interview set was slightly smaller than the goal (n= 163). The 
sample was also designed to be representative of client, collaborator, and participant subgroups as was the 
previous survey sample. Refer to Appendix B for a version of this year’s instrument. 

The client interview process presented both strengths and challenges. The evaluators found that it was 
valuable to talk with clients about their experiences in a detailed manner and about impacts on their work. 
The themes that emerged from the interviews provide a telling story of what the Network has 
accomplished over the past five years. The interviews yielded a rich data set with responses that were 
more thoughtful and in-depth than those collected via the mail surveys of previous years. Moreover, it 
was easy to see how the responses correlated to the performance indicators. However, because the process 
was more complex than initially conceived, and evaluators have different levels of experience, skill, and 
comfort in doing this kind of task, a number of issues were encountered. 

With regards to the sampling procedures, it was challenging to incorporate both purposiveness and 
randomness into the small samples (n=16 per consortium), while addressing multiple criteria and 
considerations. The participating organizations varied somewhat in their implementation of the sampling 
procedures. In addition, because the CCDDS is not directly linked to a client database, research was 
required to obtain some client contact information that was not readily available; identification of names 
was especially a problem for the one consortium that was not refunded and whose funding period had 
ended by the time the interviews were conducted. In spite of these limitations, the sampling procedure 
produced a diverse mixture of clients who are familiar with our work and provided a range of 
perspectives. 
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The interview design, much of which consisted of forced choices followed by open-ended prompts and 
probing for examples, was structured with the goal of obtaining explicit answers that were correlated to 
the performance indicators. This level of standardization probably also helped reduce interviewer bias 
across the many interviewers since interviewer training was not provided. The protocol guidelines, use of 
a common protocol, and common cover letter format were deemed helpful. However, the protocol was 
considered by some evaluators to be excessively standardized and long, and somewhat awkward to use. 
The forced choices seemed to place too much emphasis on the quantification of results. Some interviews 
went smoothly, whereas for others, the protocol caused redundancy and respondent fatigue while also 
making it difficult for the clients to tell their stories in a natural way. A bias toward positive responses 
was found; this was perhaps influenced by the protocol’s structure, and/or possibly by the nature of the 
interview administration — some clients may have been hesitant to provide negative feedback in such an 
interpersonal, one-on-one situation. It was found that the richest data was yielded by Section D of the 
interview, which contained open-ended questions. 

Although it was somewhat helpful to use the customized Excel and Filemaker Pro templates for entering 
data, the process of entering interview data was found to be, at the same time, cumbersome and time- 
consuming due to software limitations. In addition, there were variations among the participating 
organizations with regards to the way the data was captured-some recorded responses in greater detail. 
The structure of the protocol also led to difficulties in coding and analysis of the open-ended data. The 
coding process was found to be tedious and not well defined; some codes were redundant or not 
applicable, and some themes that appeared in the data were not addressed in the codes. The committee 
found the perspective from the external expert during the quality assurance review process to be helpful 
and informative. 

A number of suggestions were generated for improving the methodology for next time. If the research 
design was developed further ahead of time, more flexibility could be incorporated into it, and challenges 
vis-a-vis sampling could be shared and/or sampling plans audited. The development and use of a client 
database linked to the CCDDS would ease the process of obtaining contact information. It is also 
worthwhile to consider using a less structured interview protocol with fewer questions, and possibly focus 
on gaining data from each client about the particular context, process (nature of their involvement), and 
three greatest impacts. The protocol might also ask open-ended questions first, instead of forced-response 
questions, thereby giving the interviewer more flexibility and allowing themes to emerge. To facilitate the 
interviews and provide a better shared understanding of qualitative data analysis, professional 
development should be provided for evaluators that would familiarize them with the protocol while 
focusing on strategies for administering the interviews and techniques for analyzing the open-ended data. 
The consistency/reliability of coding would also be enhanced by developing overarching codes for the 
entire protocol as a whole group, instead of developing section-specific codes in small groups. 

Despite these limitations, the CCDDS and the client interview together comprise a valuable source of 
information for consideration by Network staff members, sponsors, and partners. The CCDDS 
documents the work of the consortia and clearinghouse at a program level. The interview provides useful 
indications of the quality and impact of consortia work as reported by clients as well as the lessons 
learned during the grant period. The data reported through the interview suggest that it will be an 
important tool to go beyond the information in the CCDDS and enhance internal evaluation efforts 
overall. 
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E=MSC 2 Client Interviews: 

What can we say about our sample? 

SAMPLE SIZE AND GEOGRAPHIC REPRESENTATION 

• The final overall sample size is 1 63 clients. 

• Each organization’s sample size ranges from 8 to 17, with a mean of 14.8. 

• All but two of the organizations completed interviews with at least 16 clients. 

• All 50 states plus 7 territories/districts/possessions are represented in the aggregated sample. 

LONGEVITY: HOW MANY (NUMBER/PERCENT) OF OUR CLIENTS WERE INVOLVED FOR 
MORE THAN 1 YEAR? (QA5) 

• Among the 163 clients interviewed, the first year of involvement ranges from 1981 (n=l) to 2000 
(n=3), with 48% (78) of the clients becoming involved in 1995, 1996, or 1997 (the most frequently 
cited years, overall). 

• 91% (149) of the clients became involved in 1998 or earlier; 82% (133) in 1997 or earlier. 

• For each organization, the median response ranges from 1993 to 1998, with all or most of each 
consortium’s clients having been involved since 1998 or earlier. 

AT-RISK: HOW MANY (NUMBER/PERCENT) OF THE SCHOOL DISTRICT-AFFILIATED 
RESPONDENTS WORK/WORKED WITH MAJORITY AT-RISK POPULATIONS? 

Of the 71 school district-affiliated clients, the schools or districts of 34 of them (51.5% of the 66 valid 
cases) have 50% or more students who are eligible for free or reduced-price lunch. The schools or 
districts of 33 clients (52.2% of the 69 valid cases) have a majority of students who are Title I, Native 
American, Pacific Islander/Native Hawaiian, or LEP. 

AFFILIATIONS: HOW MANY DIFFERENT AFFILIATIONS ARE REPRESENTED IN OUR SET 
OF RESPONDENTS? (QA2) 

• The number of affiliations represented in each organization’s data set ranges from 2 to 7, with a mean 
of 5.1 

• The affiliation that is represented by the most organizations (all 1 1 of them) is State Agency/Level. 
The least well distributed categories include Professional Association and Federal Agency/Level, 
each of which is represented by 2 organizations. 

• Overall, the employer category with which the greatest number of clients is affiliated is Public School 
District/Building (65, 40%). When combined with the Private School District/Building category, the 
number of clients affiliated with a school district increases to 44% (71) of the overall sample. 

ROLES: HOW MANY DIFFERENT ROLES ARE REPRESENTED AMONG OUR SCHOOL 
DISTRICT- AFFILIATED RESPONDENTS? (QA3a) 

• The 71 clients affiliated with a school district were asked to identify their primary role. The number 
of school-district-affiliated roles represented in each organization’s data set ranges from 0 to 6, with a 
mean of 3.6. (Two organizations’ samples did not include any school-district affiliated clients.) 

• The district-affiliated role that is represented by the most organizations (appearing in 9 consortium 
data sets) is Curriculum/Content Specialist. The least well distributed role, aside from the Other 
Teacher” and “Other” categories, is Middle/Junior High Teacher, appearing in 5 consortium data 
sets. 

• Overall, the roles most commonly cited by school-district-affiliated clients include 
Curriculum/Content Specialist (22, 31%), Administrator (15, 21%), and Elementary Teacher (13, 
18%). 
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DISCIPLINE: HOW MANY (NUMBER/PERCENT) OF THE RESPONDENTS WERE MATH 
PEOPLE? SCIENCE PEOPLE? BOTH? (QA1) 

Of the 163 clients who were interviewed: 

• 78, or nearly half (47.9%) consider their work to be related to both math and science education; 

• 21.5% (35) consider their work to be related specifically to math education; 

• 28.8% (47) consider their work to be related specifically to science education; and 

• 1 .8% (3) were unsure. 

Within each organization’s data set, too, the math and science disciplines are, in general, fairly evenly 
represented. 
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Appendix B 

E=MSC 2 Client Interview Protocol 



Date: 


ID Region/Interview #: 02 / 


Consortium: 


Interviewer 


Interviewee Name: 


Title: 


Organization 


State Abbv: (location of client’s school 


district or organization) 


A. Hello, this is 


. I’m with the Mid- Atlantic Eisenhower Consortium. Thank you for 



taking the time to talk with me today. 



B. As you know, this interview will help the Consortium to measure the impact of our two main 
types of activities: technical assistance (including professional development) and networking and 
collaboration. Your responses will be used to help the consortium plan future high-quality 
activities that are relevant to the local, state, and regional needs of its clients. 

C. Your responses will be kept confidential and anonymous, and they will be summarized with the 
responses from other clients and analyzed in a report to the U.S. Department of Education and 
Congress. 

Yes 1 

No 2 

D. Okay, let's begin. 



Interview start time: 
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A. BACKGROUND INFO 

(** FOR ALL RESPONDENTS **) 



In this first set of questions, I’d like to confirm your affiliation, as well as the nature of your involvement 
in consortium activities. 

QA1. My first question is, do you consider your work Math 1 

to be related to science education, math Science 2 

education, or both? Both 3 

NS/DK 9 

RA 7 

School district/building: Private 02 

QA2. I understand that your primary employer or (IF ANY OF THE BELOW, 

affiliation is_NJIT. Is that correct? GO TO QA2a) 

Institution of Higher Education 03 

(NOTE: Confirm the client’s role, as you know Professional Association 04 

it, using the survey options as follows. If your Business/Industry 05 

records are incorrect, identify the correct role by Informal Science 06 

reading the list of affiliations.) Community member or parent 07 

Federal Agency/Level 08 

State Agency/Level 09 

QA2a.(7F NOT WITH A SCHOOL DISTRICT Intermediate/County Education 

AND NOT A PARENT, COMPLETE THE Agency/Level 10 

FOLLOWING QUESTIONS THEN SKIP TO ENC 1 1 

QA5) National Science Foundation 

(SSI/USI/RSI/LSC) 12 

In what state or state(s) do you work? Regional Service Provider 

(RTEC/Lab/etc) 13 

Other (SPECIFY) 14 

In a sentence or two, can you tell me briefly 

about your organization and what you do? RA 77 



School district/building: Public 01 

Teacher: Elementary 11 

QA3a. (IF SCHOOL DISTRICT AFFILIATION) Teacher: Middle/Jr. High 12 

Teacher: High School 13 

I also understand that your primary role is that Teacher: Other (e.g., K-12) 14 

of a . Is that correct? Curriculum/Content Specialist (e.g., math 

supervisor, science resource tchr) 21 

Administrator (e.g., principal, vice 

principal, superintendent) 31 

Other (SPECIFY) 41 



NA 88 

RA 77 
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QA4a-l . Approximately how many students are 500 or less 1 

enrolled in your school ? 501-1000 2 

1001-1500 3 

(DO NOTREAD LIST) 1 50 1 -2000 4 

2001 or more 5 

NS/DK 9 

NA 8 

RA 7 

QA4a-2. Approximately how many students are 2000 or less 1 

enrolled in your district ? 2001-4000 2 

4001-6000 3 

(DO NOT READ LIST) 6001-8000.... 4 

8001-10,000..... 5 

10,001 or more 6 

NS/DK 9 

NA 8 

RA 7 

QA4b-l . What percentage of students in your school Less than 35% 1 

are eligible for free or reduced-price lunch? 35-49% 2 

50-74% 3 

(DO NOT READ LIST) 75% or more 4 

NS/DK 9 

NA 8 

RA 7 



QA4b-2. What percentage of students in your district Less than 35% 1 

are eligible for free or reduced-price lunch? 35-49% 2 

50-74% 3 

(DO NOTREAD LIST) 75% or more 4 

NS/DK., 9 

NA 8 

RA 7 
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QA4c-l . Which of the following categories describe the 
majority of students in your school ? 

(READ LIST) 



QA4c-la. Title I 
QA4c-lb. Native American 
QA4c-lc. Pacific Islander/Native Hawaiian 
QA4c-ld. Limited English Proficiency 
QA4c-le. Other special populations (SPECIFY): 



QA5. In approximately what year did you first become 
involved in a consortium-supported activity? 

Year 

NS/DK 9999 

RA 7777 



QA6a. As you know, the Mid-Atlantic Eisenhower 
Consortium provides professional development, 
technical assistance, and resources to promote systemic 
improvements in math and science education. 



The Consortium also promotes networking and 
collaboration among local, state, and regional 
organizations. By regional, we mean within the multi- 
state region that the Consortium serves. 



I understand that you have been involved in (READ LIST 
of key consortium activities and/or projects that the client 
was involved in over the past 5 years), which have been 
supported, in part, by the Mid-Atlantic Consortium. Is 
this correct? 



Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 



QA6b. .Were you involved in any other key 
consortium activities in the past 5 years? 

(IF YES) Please list those activities for me. 



Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 
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Q A6c . INTER VIE WER: Based on your records and on 
the client ’s response to QA6a and b, confirm 
which sampling criteria the client satisfies. If you 
are unsure of the client 's status, confirm the 
categories with the client. 



CRITERION 1: Teacher, administrator, and/or provider 
of professional development who 
participated in the Consortia’s continuing 
technical assistance. 



CRITERION 2: Participant in one of more Consortia 
partnerships, collaborations, teams, 
and/or networks. 



Criterion 1 

(GO TO PART B, PG 6) 

Criterion 2 

(GO TO PART C, PG 10) 

Both 

(GO TO PART B, PG 6) 
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B. TRAINING AND TECHNICAL ASSISTANCE 



(**FOR RESPONDENTS SATISFYING SAMPLING CRITERION #1**) 

My next few questions relate to your experiences with the professional development and/or technical 
assistance activities sponsored by the the Mid- Atlantic Eisenhower Consortium. I’d like to ask you 
several questions about the benefits and other outcomes related to your participation in these activities. 

Earlier you stated that you had been involved with the Consortium since [**see QA5**] When 
answering these questions, please reflect back over the years that you have been involved with the 
consortium’s professional development and/or technical assistance activities. 

I’m going to read a list of possible outcomes related to your involvement in professional development 
and/or technical assistance activities sponsored by the Consortium. After I read each statement , I d like 
you to tell me whether or not it is true for you. After we’ve gone through the list of outcomes, I will go 
back and ask you to give me one example of each outcome you mentioned. Let’s begin. 



QB7a. Did the professional development and/or 
technical assistance you received. . . . 

. . . .assist you in implementing or helping others 
to implement curriculum aligned with National 
or State standards? 



No 2 

NS/DK 9 

NA 8 

RA 7 



Yes 1 

QB7b. Did it assist you in implementing or helping 
others to implement instructional practices to 
attain National and/or State standards? 



Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 
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QB7c. Did it assist you in implementing or helping Yes 1 

others to implement assessment aligned with No 2 

National and/or State standards? NS/DK 9 

NA 8 

RA 7 



QB8. Did the professional development and/or Yes 1 

technical assistance enable you to improve No 2 

instructional practices or help others improve NS/DK 9 

their instructional practices in math and/or NA 8 

science? RA 7 



QB9. Did it help to meet the needs of at-risk, under- Yes 1 

represented, and/or underserved students in No 2 

math and/or science? NS/DK 9 

NA 8 

RA 7 



QB10. Did it enable you to improve student Yes 1 

engagement in math and/or science? No 2 

NS/DK 9 

(NA, IF CLIENT IS NOT A TEA CHER; REFER NA 8 

TOQA2) RA 7 



QBlO-e. (EXAMPLE/CLARIFICATION 
-see QB 1 3) 
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QB11. Did it enable you to enhance student Yes 1 

performance in math and/or science? No 2 

NS/DK 9 

(NA. IF CLIENT IS NOT A TEACHER; REFER NA 8 

TOQA2) RA 7 



QBll-e. (EXAMPLE/CLARIFICATION 
-see QB13) 



QB12. Can you think of any other important outcomes Yes 1 

resulting from your involvement in this No 2 

professional development and/or technical NS/DK 9 

assistance, whether they are positive or NA 8 

negative? RA 7 



QB12-a. (IF YES) Please describe. 



QB13. You’ve said that your participation in this professional development and/or technical assistance 
has led to a variety of outcomes. Now, for each outcome you noted, I’d like to go back and ask 
you to give me one example that illustrates that outcome. In each example you provide, please be 
specific in describing the way in which the outcome affected you, students, and/or others. I will 
also ask you to clarify any outcomes that you were unsure about. 

(INTERVIEWER: GO BACK TO ITEMS #7-11, and for each outcome that the respondent 
experienced or was unsure about, ASK FOR EXAMPLES OR CLARIFICATION. Also be sure 
to probe for the level of impact, such as whether it impacted teachers, students, the organization, 
etc.) 

(SAMPLE PROBE FOR “YES” RESPONSES: “You stated that your involvement in the 
[**Consortium’s**] professional development and/or technical assistance activities helped to 
improve instructional practices. Can you tell me whose instructional practices were improved? In 
what ways have instructional practices changed as a result? How have these changes benefited 
you, students, or others? Please give specific examples.”) 

(SAMPLE PROBE FOR “NOT SURE” RESPONSES: “You stated that you were unsure whether 
your involvement in the [**Consortium’s**] professional development and/or technical 
assistance activities helped you to improve instructional practices. Can you please explain why 
you were unsure?”) 
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QB 14. Of all the outcomes you described as a result of Curriculum alignment 01 

your involvement in this professional Instructional alignment 02 

development and/or technical assistance, which Assessment alignment 03 

one outcome was the most significant, in your Improve instruction 04 

opinion? Meet needs of at-risk, etc 05 

Improve student engagement 06 

QB 1 4a. Why was it the most significant? Enhance student performance 07 

Other (SPECIFY) 08 



NS/DK 99 

NA 88 

RA 77 



QB15. Did this professional development and/or Yes 1 

technical assistance meet your expectations? No 2 

NS/DK 9 

NA 8 

QB15a. In what ways [DID IT]/[DID IT NOT] RA 7 

meet your expectations? 



If client also satisfies Criterion 2 

(i.e., participates in one or more Consortium 

partnerships, collaborations, 

teams and/or networks): GO TO PART C, PG 10 

If not: GO TO PART D, PG 14 
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C. COLLABORATION AND NETWORKING 



(**FOR RESPONDENTS SATISFYING SAMPLING CRITERION #2**) 

This set of questions relates to the nature of and outcomes related to your collaboration and networking 
with the consortium. First, I’d like to ask you about the nature of your collaboration with the 
[**Consortium**]. 

Earlier you stated that you had been involved with the consortium since [**year**]. When answering the 
following questions, please reflect back over the years that you or your organization has collaborated 
with the consortium. 



QC16a. Can you briefly describe the nature of this collaboration? For example, please describe whether 
this collaboration involved planning, product development, provision of space or staff time, or 
other things. 

(DESCRIBE BELOW) Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 



QC 16b. Did different organizations contribute different Yes 1 

things? No 2 

NS/DK 9 

NA 8 

RA 7 

QC16b-e. Can you give me an example? 



Now, I’m going to read a list of possible outcomes related to your collaboration with the 
[**Consortium**]. After I read each statement. I’d like you to tell me whether or not it is true for you. 
After we’ve gone through the list of outcomes, I will go back and ask you to give me one example of 
each outcome you mentioned. Let’s begin. 

QC17a. Did your collaboration with the 

[**Consortium**] strengthen relationships? 

QC17a-e. (EXAMPLE/CLARIFICATION 
-see QC18) 
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Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 



QC 1 7b. Did your collaboration with the 

[**Consortium**] increase coordination 
in providing services? 

QC17b-e. (EXAMPLE/CLARIFICATION 
-see QC 1 8) 



Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 



QC17c. Did it increase access to resources? 

QC17c-e. (EXAMPLE/CLARIFICATION 
-see QC18) 



Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 



QC17d. Did it leverage resources and efforts for greater 
impact? 



QC17d-e. (EXAMPLE/CLARIFICATION 
-see QC18) 



Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 



QC17e. Did your collaboration with the 

[**Consortium**] inform policy decisions 
at any level? 

QC17e-e. (EXAMPLE/CLARIFICATION 
-see QC 1 8) 



Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 
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QC17f. Did it assist you in carrying out your work more 
effectively? 

QC17f-e. (EXAMPLE/CLARIFICATION 
-see QC 1 8) 



Yes 1 

No 2 

NS/DK 9 

NA 8 

RA 7 



QC17g. Can you think of any other outcomes resulting 
from your collaboration with [**Consortium**], 
whether they are positive or negative? 

QC17-g. (IF YES) Please describe. 



Yes 1 

No 2 

NS/DK . ...9 

NA -.8 

RA 7 



QC18. You’ve said that your collaboration with [**Consortium**] has led to a variety of outcomes. 

Now, for each outcome you noted, I’d like to go back and ask you to give me one example that 
illustrates that outcome. In the examples you provide, please be specific in describing the way in 
which each of these outcomes affected you, your organization, or the schools you work with. I 
will also ask you to clarify any outcomes that you were unsure about. 

(INTERVIEWER: GO BACK TO ITEMS #17a-f, and for each outcome that the respondent 
experienced or was unsure about, ASK FOR EXAMPLES OR CLARIFICATION. Also be sure 
to probe for the level of impact, such as whether it impacted the respondent, his/her organization, 
the schools with which the respondent works, etc.) 

(SAMPLE PROBE FOR “YES” RESPONSES: “You stated that your collaboration with the 
[♦♦Consortium**] strengthened relationships. Between what people, groups, or organizations 
were relationships strengthened? In what ways? How did this benefit you, your organization, or 
others with whom you work? Please give specific examples.”) 

(SAMPLE PROBE FOR “NOT SURE” RESPONSES: “You stated that you were unsure whether 
your collaboration with the [**Consortium**] helped to strengthen relationships. Can you please 
explain why you were unsure?”) 



O 

ERIC 



FINAL VERSION 



9/1 1/00 



B-12 



79 



QC19. Of all the outcomes you described as a result of Strengthen relationships 01 

your collaboration with the consortium, which Increase service coordination 02 

one outcome was the most significant, in your Increase access to resources 03 

opinion? Leverage resources/efforts 04 

Inform policy decisions 05 

QC19a. Why was it the most significant? Help you work more effectively 06 

Other (SPECIFY) 07 



NS/DK 99 

NA 88 

RA 77 



QC20. Did your collaboration with the consortium Yes 1 

meet your expectations? No 2 

NS/DK 9 

QC20a. In what ways [DID IT]/[D1D IT NOT] NA 8 

meet your expectations? RA 7 



QC2 1 . Has your collaboration with the consortium Yes 1 

provided you, your organization, your state, No 2 

and/or teachers or students with benefits that NS/DK 9 

would not have been afforded them otherwise? NA 8 

RA 7 

QC21a. Please explain. 



GO TO PART D, PG 14 
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D. OVERALL RATING OF THE CONSORTIUM 



(**FOR ALL RESPONDENTS**) 

The next few questions focus on what you perceive to be the consortium’s role and the quality and 
impact of its work overall. 

QD22. In general, what niche do you feel the consortium fills in your region, if any? 



QD23. Would you say that the consortium addressed Yes 1 

any needs that would have been difficult or No 2 

impossible for another organization to address? NS/DK 9 

NA 8 

QD23a. Please explain. RA 7 



QD24. Overall, when you reflect on the Consortium’s 

role, what other impact or impacts, if any, did its 
work have on mathematics and/or science 
education? 



QD24a. (IF ADDITIONAL IMPACT IS 
MENTIONED BY CLIENT) 

In what ways do you feel the 
Consortium’s work contributed to this 
impact? 
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QD25. Is there any way in which the services and/or Yes 1 

products of the Consortium have not met your No 2 

expectations? NS/DK 9 

NA 8 

(EF YES) In what ways did they not meet your RA 7 

expectations? 



(EF NOT SURE) Please explain. 



QD26. How can the Consortium improve its services and/or products in the future, to better meet either 
your own professional needs, or the needs of your organization, school, district, state or region? 

(PROBE to determine at what level this need exists — professional, organizational, school, 
district, state, or regional level) 



QD27. Would you like to share any additional Yes 1 

comments, either positive or negative, about No 2 

[**Consortium**], its services, and/or products? NS/DK 9 

NA 8 

QD27a. (IF YES or NOT SURE) What RA 7 

additional comments do you have? 



THESE ARE ALL THE QUESTIONS I HAVE FOR YOU. 
THANK YOU FOR YOUR TIME. 



Interview ending time: 



Q 0 
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Audit Report on Interview Methods 



ROSENBLUM BRIGHAM ASSOCIATES 

220 West Rittenhouse Square, #16D • Philadelphia, PA 19103 • (215) 735-4159 • FAX (215) 735-5767 



1/12/01 

To: Keith Kershner et aJ. 

From: Sheila Rosenblum 
Re: Review of Coding 



Introduction 

This is a memorandum/report of findings from an audit of the coding of open-ended 
responses in interviews conducted in each of the Mathematics/Science Regional Consortia. 
Interviews were conducted in each of the ten regions and by the ENC. Each of the 11 
organizations conducted between 14 and 17 interviews. The audit was based on a review often 
sets of interviews; materials from one region arrived too late to be included in the audit process. 
The purpose of the audit was to review a sample of coded responses to 32 open-ended questions, 
and to assess the quality of the coding process. 



The Audit Process 

The following procedures were used to audit the coded responses: 

1) A sample of 10% of the responses (codes) for each open-ended question, in each of 
the sections B,C,and D were reviewed, with the modifications noted below. 

2) At least one response for each consortium for each question was included, even if it 
didn’t fall in the 10% sample. 

3) For questions where there were much fewer responses than others, 1/8 rather than 1/10 
of the responses were reviewed. 

4) For each question, the times the code was found to be acceptable was counted; and the 
times there was disagreement, or the code was questioned was counted. The second group 
(questions or disagreements), included instances where the code was disputed (seen as incorrect), 
or where the code(s) were fine, but were not exhaustive enough, and there important codes left 
off. The percentage it codes where there was agreement, vs. questioned or disagreed was 
calculated for each of three sections (B. C, and D). However, the percentage of disagree vs. 
non-exhaustive was not calculated. 



Findings 

In conducting the audit, several factors became evident: 
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1) Descriptions of examples (the open-ended responses) varied greatly in detail and in 
length, ranging from a phrase, to a sentence, to several sentences, to a paragraph, to 1/2 page 
to a full page. 

2) These probably varied in part by the nature of the interview (some probed more 
intensely than others, as is documented in the transcription), and the nature of the interviewer’s 
recording. 

3) Interviews were in some cases transcribed and not edited. Some are hard to 
understand because words were not clear, or not heard well, or misunderstood. Also, they were 
not edited to make more comprehensible to a reader. 

4) In some cases, the interview responses did not make sense, were somewhat gibberish, 
or were possibly placed as a response to a wrong ‘question. 

5) Some coders were more exhaustive in their coding than others, i.e., including several 
appropriate response codes in a paragraph rather than one or just a few. 

6) Different committees developed codes for each of the three sections of the interview, 
sections B, C, and D. It seemed liked similar concepts were coded or treated differently across 
the three sections. Some gave clearer definitions than others. This may have added to the 
confusion and possible frustration with the task. 

7) It seemed likely that there were inconsistencies in how coders interpreted the meaning 
and definition of some codes. This may have been exacerbated by the possibility that there was 
overlap and potential duplication in some of the codes, (or unclear definitions emphasizing the 
differences between them) and so coders varied in the codes that they chose. For example, B-06 
and B-08, C-13 and C-15 among others, had overlapping definitions; Further, it was sometimes 
also unclear (to the coder and to the reviewer) whether the code referred to a condition 
facilitating the successful outcome (e.g., facilitating the outcome of collaboration) or whether it 
was the subsequent outcome (e.g., the outcome of collaboration). Thus the codes may mean 
different things to different coders. 



Coding for a sample of responses to each question was reviewed within each section (B, 
C, and D). The percent ot the reviewer’s agreement with the codes was approximately 65%, and 
was very similar across the three sections. Approximately 35% of the time, the reviewer either 
disagreed with the code, or felt that additional codes were also indicated, i.e., the coding was not 
complete enough. 

All of these issues, especially the lack of standardization in approach to the coding and 
the levels of completeness ot the examples had implications for the quality, scope, and potential 
validity and reliability of some of the coding. 
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Implications and Recommendations 



The implications of these findings depend on the ways in which the codes will be used. 
If the codes were to be used in a quantitative analysis, the issues outlined above and the level of 
reliability of the codes appears troublesome, and this reviewer would have serious reservations 
about the credibility of such quantitative analyses. However, if the codes were to be used to 
select potential narrative examples to illustrate a finding or outcome, or to select a particular 
interview for review for use in a vignette, then the level of reliability is less serious. The analyst 
or writer could select, and if the example is not so illuminating, one could select other examples! 

The following recommendations are offered if the consortia choose to code narrative 
responses in the future: 

• Be more specific about the definitions of the codes, and review for redundancy 
and overlap; 

• Check for consistency of codes across sections; 

• Conduct careful training sessions for codings, to insure understanding and 
consistency; 

• Conduct a reliability check of the codes for a sample of responses. Have two 
people code the same sample of responses, and have them discuss and reconcile 
their differences, if any. This could also be part of the training. 

Please call if you have any questions about this memo. 



Appendix D 



Date: 



MID-ATLANTIC EISENHOWER CONSORTIUM 
ACTIVITY DESCRIPTION FORM 



1. Activity Title: 

2. Activity Description: 



3. Staff Liaison: 

4. Client contact (if applicable) Organization: 

Client phone/email: • : 1 — 

5. Dates of this activity: to . 

6. States served (check all that apply): DE Q , DC Q , MD Q , NJ Q , PA Q , Region-wide 3 , Nation-wide Q 

7. Location of Activity: 

8. Is this a continuing activity: Yes □ No □ If yes, 

Is this the first report? YesQ NoO 

Date the continuing activity began: 

Date the activity is expected to conclude: 



Major Service Cateeorv (You can rank up to 3) 


1 1 . Contact Methods (Check all that apply ) 


□ 


Training 


□ 


In-person 




check here if training is for trainers 


□ 


Telephone 


o. 


Technical assistance 


□ 


E-mail/electronic (web, listserv) 


□ 


Information dissemination (see Publications List) 


□ 


Print 


□ 


Network building 


□ 


Audiotape 


□ 


Other (describe) 


□ 


Videotape 






□ 


CD-ROM 


Content Focus Area (You can rank up to 3) 


□ 


Other computer disks 


□ 


Assessment 


□ 


Teleconference 


□ 


Standards 


□ 


Other (describe) 


□ 


Curriculum frameworks 






□ 


Programs and curricula 


12. Intensitv 


□ 


Professional development 


Number of hours ( 1 day=6 hours) 


□ 


Technology 


Total Number of participants 


□ 


Collaboration and communication 






□ 


Equity 






□ 


Community Outreach 






□ 


Evaluation 






□ 


Other (describe) 







13. What did the Consortium add or contribute to the activity (value added)? 



O 
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(over) 

D-l 



8S 



Mid-Atlantic Eisenhower Consortium 



14. Participant Affiliation (Number of each) 

Check box if collaborator ▼ and specify organization names : 



Federal agency level 




□ 


SEA/state agency level 




□ 


Intermediate agency level 


□ 


Local district (public) 




□ 


Local district (private) 




□ 


IHE 




□ 


Professional association 




□ 


Business/industry 




□ 


ENC 




□ 


Regional service providers \ 


□ 


NSF 




□ 


Informal science entity 




□ 


Community members/parents \ 


□ 


Other 




\ a 


Unknown 

TOTAL, must equal participants 





15. Breakout of Local District, Total = 

Level Early childhood 

Elementary 

Middie/Jr. High 

Senior High 

District level 

Other 

Unknown 

Total * 

Role Administrator 

Teacher 

Curriculum specialist 

Other 

Unknown 

Total * 

At-risk Yes 

No 

Unknown 

Total * 

*All totals must be equal 



Type of at-risk environment (check all that apply) 

□ Qualify for free or reduced price lunch 

□ Title 1 

□ Native American 

□ LEP/ELL 

□ Rural ' 



16. What were the outcomes of the activity? How will participants and their students benefit? 



(November 2, 2000) 
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Appendix E 

OERI Performance Indicators for 2000 

Quantitative indicators of performance related to consortia and clearinghouse activities have been 
under development by the U.S. Department of Education for the last few years. Network staff 
have had opportunities for input into the development process. A set of key indicators was 
initially submitted in the Department’s overall FY 1999 Annual Plan. Subsequently, the set was 
expanded, conveyed to the Network, and later refined by the Department with input from the 
Network. 

While the Performance Indicators are still subject to revision, and were not designed specifically 
to match Network evaluation efforts, the data reported herein provide some results that speak to 
the indicators. This section presents an overview of results selected to represent performance in 
the 2000 fiscal year across indicators. 

1. Overview. Table 4 gives the overview across indicators. On the 1 1 indicators presented 
here, the benchmark standards were met or exceeded for 9. Four of these benchmarks utilized 
data for 2000 from the CCDDS: 1.2, 1.7, 1.8, and 2.1. Measurement of indicator 1.5 on student 
test scores has not yet been implemented. The benchmark of 60 percent intensive activities for 
indicator 1.2 was not met, as only 51 percent of activities met the benchmark. Measurement of 
the remaining six indicators is dependent on quantitative data obtained from clients. Such data 
were not available for 2000 because the client interviews were not designed to yield data that 
would meet reliability and validity standards for quantitative analysis. These interviews were 
designed for in-depth qualitative use. Thus, the results reported for these indicators in Table 4 
was taken from the 1999 Client Survey data, which has also been reported for 1999. 

2. Individual Indicators. Each Performance Indicator is described below, along with the 
benchmark for desired performance and the pertinent results derived from the CCDDS and 
Participant Survey. 

• 1.1 — Alignment with standards. At least 80% of participants in consortia technical assistance 
activities (including training) will report that the content is explicitly aligned with national or 
state standards and/or is focused on assisting in the implementation of standards. Results : 
Over 98% of survey respondents indicated in 1999 that the content of training or technical 
assistance was moderately or extensively aligned with standards. (See Table in June 28, 2000 
report.) 

• 1.2 — Intensity of technical assistance. At least 60% of consortia technical assistance 
(includes training) will be 12 hours or more in duration. Results : 51% of CCDDS-reported 
activities were 12 hours or more in length (Table 3). 

• 1.3 — Improvement in participants 'practice. At least 80% of the teachers, administrators, and 
providers of professional development who participate in the consortia’s continuing technical 
assistance will report improvement in their practice. Results : Over 95% of survey 
respondents indicated in 1999 that the training or technical assistance received was 
moderately or extensively useful in improving their instructional practice. (See Table 16 in 
June 28, 2000 report.) Note: Respondents represent all technical assistance; the “continuing” 
subset was not representative of the CCDDS-reported total of participants involved with 
intensive technical assistance or training. 
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Performance Indicator Results for 
October 1999-September 2000 





Indicator 


Standard 


Data 

Source 


Results Met 


1.1 


TA* aligned with standards 


80% of participants report 


Survey 


99% 


✓ 


1.2 


TA intensity 


60% of activities are 
12+ hours 


CCDDS 


51% 


X 


1.3 


Intensive TA improvements in 
practice** 


80% of participants report 


Survey 


96% 


✓ 


1.4 


Intensive TA improvements 
in student performance** 


80% of participants report 


Survey 


94% 


✓ 


1.5 


Intensive sites’ improvements 
in student scores 


Measurable improvement 


School or 

district 

assessments 


NA 


NA 


1.6 


Training of trainers produces 
training of others 


80% of participants report 


Survey 


83% 


✓ 


1.7 


Intensive TA targeted on at-risk 70% of participants report 


CCDDS 


81% 


✓ 


1.8 


All activities involve 
collaborators 


80% of all activities 


CCDDS 


90% 


✓ 


1.9 


Impact of collaboration as 
value added 


80% of team and network 
members report 


Survey 


85%- 

93% 


✓ 


2.1 


Dissemination of resources 


10% annual increase in print 
and/or Web hits 


CCDDS 


10.4% in 
Web + print 
total 


✓ 


2.2 


Utility of products*** 


50% of participants report 


Survey 


77% and 82% ✓ 
of products 



*TA includes technical assistance and training. 

**Indicator refers only to intensive subset of activities (1.2), but results here are for all activities. 
***Results are described in terms of the percentage of Consortia and ENC products that contributed 
“moderately” or “significantly” to improving the work of recipients. 
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• 1.4 — Improved student performance. At least 80% of teachers who participate in the 
consortia’s continuing technical assistance will report improvements in student engagement 
and/or student performance. Results : At least 94% of survey respondents reported in 1999 
that the training or technical assistance received was moderately or extensively useful in 
improving student engagement and student performance. (See Tables 14 and 15 in June 28, 
2000 report.) Note: Respondents represent all technical assistance; the “continuing” subset 
was not representative of the CCDDS-reported total of participants involved with intensive 
technical assistance or training. 

• 1.5 — Improved student performance in sites that receive intensive assistance from consortia. 
Assessment scores of students who have been enrolled for at least one year in a mathematics 
and science program will show improvement. Results : During the current reporting period, 
data were not yet available for this indicator. The Network plans to collect achievement data 
through intensive site work in the Middle School Math Project and other efforts. 

• 1.6 — Participation by individuals who will assist. or train others. At least 80% of participants 
in consortia training-of-trainers activities will go on to provide professional development or 
technical assistance based on the technical assistance they received from the consortia. 
Results : Over 82% of survey respondents who participated in a training-of-trainers activity 
reported in 1999 providing assistance to others as a result. (SeeTable 13 in June 28, 2000 
report.) 

• 1. 7 — Targeted services. At least 70% of the district and school staff who participate in the 
consortia continuing technical assistance will work in districts or schools with a majority of 
students who are Title I eligible. Results : 81% of the LEA participants in intensive technical 
assistance activities indicated working in schools serving at-risk students (Table 3). (This 
percentage excludes 1,927 participants whose status was unknown; if these were included, 
the percentage would be lower, not meeting the benchmark.) 

• 1.8 — Volume of collaboration. At least 80% of consortia activities will include collaborators 
from one or more stakeholder groups in planning, product development, and/or service 
delivery. Results : 90% of all activities involved at least one collaborator (Table 3). 

• 1.9 — Impact on collaboration and networking. At least 80% of members of consortia teams 
and networks will report that value was added in one or more of the following ways: 
strengthening relationships; increasing service coordination; increasing access to resources; 
or leveraging resources. Results : In 1999, over 92% reported strengthened relationships 
among collaborators, 88% increased coordination in providing services, 92% increased 
access to resources, 87% resources and efforts leveraged for greater impact, and 92% 
collaboration assisted in carrying out the work more effectively. (See Tables 32-36 in June 
28, 2000 report.) 

• 2.1 — Dissemination. The total number of consortia contacts with customers by print and/or 
“hits” on consortia electronic sites will increase by 10% annually. Result : This indicator has 
been met and exceeded. Print contacts increased from 125,212 to 129,901, while electronic 
contacts increased from 3,328,846 to 3,684,833. Thus, there was a net combined increase of 
360,736, or 10.4%. 

• 2.2 — Utility. A majority of the recipients of consortia and ENC products and resources will 
report that they have contributed to improving their work. Result : This indicator has been 
met. A majority of the recipients of consortia and ENC products and resources reported in 
1999 that they have contributed to improving their work. As reported in Table 23 of the June 
28, 2000 report, over 77% and 82% of the products for the consortia and ENC respectively 
contributed “moderately” or “significantly” to improving the work of recipients. 
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Appendix F 

Cross Reference of Interview Vignettes with the Indicators 

National Network of Eisenhower Regional Consortia and Clearinghouse 

NOTE: In Sections III and IV of this report, each quotation, or vignette, has been assigned a unique identification number. 
These numbers range from #1 (on the first page of Section III) to #83 (on the last page of Section IV). The ED numbers are 
referenced below each vignette in the report, as well as in the following table. If a vignette provides anecdotal data related 
to multiple indicators, its ED number appears in all relevant locations in the table. (Two other vignettes — 1 and 78 — do not 
relate directly to a specific indicator and are thus not included in this table.) 





Vignette ID Numbers 


OERI Indicators 


Section III A: 
Training and 
Technical 
Assistance 


Section IIIB: 
Dissemination 


Section IIIC: 
Collaboration 
and 

Networking 


Section IV: 
Conclusions 


.Technical Assistance sM 


1.1: Alignment with standards 


6, 8, 9, 10, 19, 
23, 24, 25, 28, 
30, 40 


““ 


67 


”” 


1.2: Intensity of technical assistance 




- 


- 


82 


1.3: Improvements in participants’ practice 


2, 3, 4, 5, 6, 7, 
8, 10, 11, 12, 
13, 14,15,17, 
19, 20,21,22, 
23, 24, 25, 26, 
27, 28,30,31, 
33, 34, 36, 37, 
40, 42 




58,60 




1 .4: Improved student performance 


2, 14, 15, 16, 17, 
18,19,30,42 




- 




1.5: Improved student performance in sites that 
receive intensive assistance from Consortia. 


2, 14, 15, 18, 19 




- 




1 .6: Participation by individuals who will assist 
or train others 


21,22, 29,31, 
32,33,35,37, 
38,39,40,41,42 


““ 




““ 


1.7: Targeted services 


2, 12, 14, 15, 
19, 27,28,42 


- 


61 


81, 82, 83 


1.8: Volume of collaboration 


- 






- 


1 .9: Impact on collaboration and networking 


8,21,26,37 


56 


57,58,59, 60, 
61,62, 63,64, 
65, 66, 67, 68, 
69,70,71,72,73 


74, 75, 76, 
77, 79, 80, 
83 


Dissemination :• • VV' r^" : - l 


2.1: Volume of dissemination 


- 


44 


72 


- 


2.2: Quality of dissemination 


- 


43,44, 46, 47, 
48,49,51,53,54 


- 


- 


2.3: Utility of dissemination 


— 


43, 45, 46, 49, 
50,52,53,55,56 


71 


75 



O 

ERIC 



F-l 



91 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 

Reproduction Basis 




This document is covered by a signed ‘'Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 



EFF-089 (3/2000) 




